The case for self-hosted ML infrastructure in regulated industries

← Back to Blog

Imagine a bank that has spent fifteen years collecting transaction data on 40 million customers. Every fraud pattern, every credit risk signal, every behavioural anomaly that their models detect is embedded in that data. It is, without exaggeration, their most valuable asset after their deposit base.

Now imagine that the fraud detection model that analyses this data runs on a third-party cloud provider's GPU cluster, in a data centre outside India. The raw transaction data — names, amounts, merchant categories, geolocation signals — is transmitted to that cluster for inference, and the results are returned.

This is a common arrangement. Most of the organisations we talk to have something close to this. When we ask their CISOs whether they've assessed the risk, the answer is usually a variant of: "We know it's a problem. We haven't found a workable alternative."

This article is about the workable alternative.

Three distinct risks you may not have quantified

Regulatory risk

The RBI's Master Direction on Information Technology Governance (2023) and the DPDP Act both require that personal data of Indian residents be processed within India, and that critical financial data not be transferred to foreign jurisdictions without specific conditions being met. Sending transaction-level customer data to a US or EU cloud provider for ML inference may — depending on how your DPA reads it — constitute a cross-border data transfer that requires regulatory approval you do not have.

SEBI's cloud framework for market infrastructure institutions is similarly explicit: data classification requirements effectively prohibit sensitive market data from leaving controlled environments.

Competitive intelligence risk

This one is less discussed but arguably more immediately impactful. Your ML models are trained on proprietary data and encode proprietary signals. The model weights themselves represent years of competitive learning. When those models run on shared cloud infrastructure, you are operating in an environment where — regardless of contractual commitments — your model's behaviour, inputs, and outputs are accessible to the cloud provider's infrastructure at the hardware layer.

For organisations whose entire competitive moat is their data and their models — every NBFC, every insurance underwriter, every algorithmic trading firm — this is not a theoretical concern.

Operational dependency risk

In March 2024, a major cloud provider experienced a regional outage that lasted 7 hours. For organisations running their fraud detection, credit decisioning, and KYC models on that provider's infrastructure, the operational impact was not "slow website." It was suspended customer onboarding, paused credit decisions, and degraded fraud detection across millions of active sessions.

If your ML inference pipeline has a single-provider dependency, you don't have an ML platform. You have an ML dependency — and you will feel it the first time that provider has a bad day.

What self-hosted ML actually requires

The reason most organisations don't self-host ML is not that they've evaluated it and found it lacking. It is that the perceived complexity is high, and the cloud alternative is immediately available. Let's actually enumerate what you need:

GPU compute

Modern inference workloads for mid-size models (7B–13B parameters) run well on NVIDIA L40S or A100 GPUs. A 4-GPU server handles most enterprise inference loads with headroom. The economics are straightforward: at the usage levels of a large bank's fraud detection system, owned hardware pays for itself in under 18 months versus cloud GPU pricing. For training workloads, the math is even more favourable.

Model serving infrastructure

KServe (formerly KFServing) on Kubernetes handles model deployment, versioning, A/B testing, canary rollouts, and auto-scaling on GPU utilisation. This is the infrastructure equivalent of what Replicate or Hugging Face Inference API provides — except pointed at your own GPUs, with your data never leaving your network. The API surface is identical. The developer experience is identical. The data residency is completely different.

# Deploying a fraud detection model to your own cluster # Same interface as cloud inference APIs — different endpoint apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: fraud-detector-v2 namespace: ml-production spec: predictor: model: modelFormat: name: pytorch storageUri: s3://internal-model-registry/fraud-v2/ resources: limits: nvidia.com/gpu: "1" # Inference request — identical to any cloud ML API POST https://infer.your-dc.in/v1/models/fraud-detector-v2:predict Authorization: Bearer <vault-issued-token> # Transaction data never leaves your network

Experiment tracking and reproducibility

MLflow handles experiment tracking — every training run has a logged set of hyperparameters, metrics, data hashes, and model artifacts. When your model produces an anomalous output and your regulator asks "what version of the model was running and what was it trained on," MLflow gives you a complete, traceable answer. This matters more than most teams realise until they need it.

Observability

Production ML systems fail in ways that are different from traditional software. Models drift. Their accuracy degrades gradually. Inputs change distribution. You need monitoring that is specifically aware of ML semantics — not just "is the service up" but "is the model's output distribution consistent with its training distribution." This requires prompt tracing (for LLMs), feature drift detection (for traditional ML), and token cost attribution (to understand operational costs).

The "open source model" question

A common question: "What about open-source models like LLaMA, Mistral, or Gemma? Can we just use those on our own hardware?"

Yes. Completely. This is, in many ways, the most important development in enterprise ML of the last two years. Models that are functionally competitive with GPT-4-class capability for many enterprise tasks — document classification, entity extraction, summarisation, code generation — are now available under open licences, deployable on modest hardware, with no API call leaving your network.

A bank deploying Mistral-7B for document summarisation on a 2-GPU server processes thousands of loan application documents per day, with zero data egress, at a marginal cost that is essentially electricity. The alternative — GPT-4 API calls — is faster to implement, but carries every risk we've described above.

The implementation path

For organisations starting from scratch, the sequence that works:

Start with inference, not training. Move your highest-sensitivity model to self-hosted inference first. This validates the infrastructure and closes the most critical regulatory exposure with the least disruption.
Use KServe on Kubernetes. It handles the operational complexity of model serving — auto-scaling, versioning, canary deployments — so your ML team can focus on models, not infrastructure.
Add MLflow for experiment tracking. This is essential for regulatory audit trails and model reproducibility. It is also the foundation for responsible AI practices.
Build AI observability from day one. Token cost attribution, prompt tracing, and drift monitoring are much easier to build in than to retrofit.

For CISOs specifically: The question is not "can we afford to self-host ML?" It is "can we afford the regulatory, competitive, and operational risks of not doing so?" When you frame it that way, the economics of self-hosted infrastructure look very different.

The organisations that will have a structural ML advantage in five years are not the ones that trained the biggest models. They are the ones that built the infrastructure to run any model they choose, on their own hardware, with their proprietary data, in an auditable and regulatorily compliant way.

🤖

AravaliStack Data & AI Team

AI/ML Infrastructure Engineering · Rajendra Management Pvt. Ltd.