From Federated Learning to Sovereign Foundation Models

It started in hospitals

The constraint that shaped my research is easy to state. Patient records do not leave the building. Privacy law, institutional policy, and plain ethics all push in the same direction: an enormous amount of data exists in aggregate, locked in isolated silos, none of it individually sufficient to train the model you actually want.

Federated learning answers this by inverting the usual setup. Instead of bringing the data to the model, you bring the model to the data — each site trains locally and shares only model updates, never raw records. I’ve spent years on the hard parts that hide inside that one idea: heterogeneity across sites, personalization, privacy guarantees, and generalizing to hospitals you have never seen. I wrote about all of that in the ideas behind my research.

What I want to argue here is that healthcare was never the special case. It was the early, high-stakes preview of how AI gets built everywhere data is valuable, sensitive, and owned.

From privacy to sovereignty

“Privacy” undersells what is happening. A hospital does not just want patient data kept confidential — it wants to retain control over the data and over anything derived from it. The same is true of a bank, a government, a research consortium, or a company sitting on a corpus it considers a competitive moat. The real demand is sovereignty: keep the data, keep the derived model, keep the say over how both are used.

As foundation models become essential infrastructure rather than research artifacts, that demand only sharpens. Nations want models trained on their own languages and institutions without exporting the underlying corpora. Hospitals want clinical models tuned on their populations without shipping scans to a third party. The question stops being “can we centralize this data?” — the answer is no — and becomes “how do we build a capable model across owners who will never pool their data?”

Sovereign AI is federated learning’s original problem, scaled up from a single hospital network to the level of institutions and nations.

Three principles that transfer

Three ideas carry almost unchanged from privacy-preserving healthcare AI to sovereign foundation models.

Local data control. Computation moves to the data, not the other way around. The raw corpus never leaves its owner’s environment; what circulates is a model, a gradient, an adapter — never the records themselves.

Decentralized optimization. Training becomes a coordination problem across non-identical participants. The same heterogeneity that makes hospitals hard to average — different distributions, different sizes, different label conventions — reappears across institutions and regions. The machinery I built to stabilise it, like the James–Stein shrinkage in FedStein or the cross-client feature alignment in FedAlign, is exactly the kind of tooling sovereign training needs.

Secure collaboration. Keeping raw data home is a starting point, not a guarantee — updates can leak. Differential privacy and secure aggregation turn “the data stays here” into something you can actually reason about. In FedHypeVAE we went a step further, sharing differentially private embeddings rather than raw model updates, so the collaboration carries a formal privacy budget with it.

Adapting big models without moving data

The obvious objection: foundation models are enormous, and federation already strains under the cost of shipping ordinary models around. You cannot round-trip a hundred billion parameters across a consortium every training step.

You don’t have to. The shift that makes federated foundation models practical is parameter-efficient adaptation — freeze the giant backbone and learn small, cheap-to-communicate pieces on top: adapters, low-rank updates, prompts. Each owner adapts the shared model to its own data and exchanges only the adapter. My work on Sequential Compression Layers targets exactly this regime, shrinking what foundation-model clients need to exchange during federated fine-tuning, and in BiPrompt we adapt vision-language models through prompts rather than full weights.

The update is the product

Once the adapter, not the data, is the unit of collaboration, the entire system reorganizes around moving and combining updates well. How do you aggregate adapters from owners whose data looks nothing alike? How do you stop a few large participants from dominating the result? How do you keep the rare, important signal — the equivalent of the rare diagnosis — from being averaged into oblivion?

These are the questions I’ve been circling for years in a clinical setting. In FEDTAIL we used sharpness-guided gradient matching so rare classes survive aggregation; in FedVR a variance-regularized hypernetwork keeps any single domain from dominating. At foundation-model scale the variables change, but the aggregation problem is recognizably the same one.

Trust as a design constraint

When a model is trained across institutions that do not fully trust each other, “trustworthy AI” stops being a slogan and becomes an engineering requirement. You need privacy you can quantify, robustness to participants who behave badly or just behave differently, and fairness across populations that are represented very unevenly in the federation.

I find healthcare clarifying here precisely because it is unforgiving: a model that is accurate on average but fails on an underrepresented group is not acceptable in a clinic, and it should not be acceptable in a sovereign model either. The constraints that medicine forced me to take seriously — calibrated privacy, fairness across cohorts, reliability on the tail — are the same constraints frontier AI is now discovering it cannot skip.

From models to agents

The newest twist is that the thing being deployed is increasingly not a single model but a system of agents — models that call tools, hold memory, and act. During my time at Samsung working on an AI operating system for managing LLM-based agents, the recurring theme was governance: isolation, scheduling, access control, and resource limits for autonomous systems running under real constraints.

Put sovereignty and agents together and you get the direction I’m building toward now: decentralized agents that coordinate across tools, data, and institutional boundaries without surrendering control to a central operator. That is the premise behind ErdosFC, a federated computing runtime, and ErdosFAI, a platform for building and governing agents across decentralized environments.

Closing thoughts

I used to describe my research as “federated learning for healthcare.” I now think the healthcare part was the proving ground and the federated part was the point. The constraints that make medical AI hard — data you cannot move, owners who keep control, privacy you must guarantee, fairness you cannot fake — are becoming the default conditions for building frontier AI at all.

If foundation models are infrastructure, then the real question is who gets to train, adapt, and deploy them, and on whose terms. I think the answer looks a lot like federation. If you’re working on this — or you see it differently — I’d love to hear from you: sunnygupta@iitb.ac.in.