Motivation for Federated Learning

"For thirteen chapters they shuffled me wherever the compute was. Then a hospital said the data stays put, and for once the model had to come visit."

A Shard That Was Never Allowed to Leave Home

Through all of Part II and the optimization chapters of Part III, one assumption was so natural that we rarely stated it: the training data sits in a store we control, and we may copy it, partition it, and ship the shards to whichever workers have spare compute. Data parallelism, the parameter server, the distributed solvers of Chapter 10, every one of them takes a central corpus and spreads it out for speed. The deciding factor was always where the compute was; the data went to meet it. Federated learning begins by denying that this is allowed. There are settings where the data physically cannot be centralized, and in those settings the entire flow of the system reverses.

Consider three of them. A keyboard app would like to improve next-word prediction from what people actually type, but the text people type is among the most private data they own, and uploading it would be both a privacy disaster and a bandwidth burden across billions of devices. A consortium of hospitals would like to train a diagnostic model on far more patient scans than any one of them holds, but patient records are bound by regulation (HIPAA in the United States, GDPR in Europe) that forbids pooling them into a shared bucket. Several banks would like a stronger fraud model trained on the union of their transaction histories, but that union is precisely the competitive and regulatory asset none of them may export. In each case the data is valuable, it is distributed by birth, and it is immovable.

1. From Sharding a Central Corpus to Training In Place Beginner

The federated setup has three moving parts and one strict rule. The parts are a coordinator (often called the server), a population of clients (phones, hospital servers, bank data centers), and a shared model. The rule is that raw data never leaves a client. A round of training proceeds as a loop. The coordinator broadcasts the current global model to a set of clients. Each selected client trains that model on its own local data for several steps of ordinary stochastic gradient descent, producing a locally improved model. Each client sends back only its update (the new weights, or equivalently the change in weights), and the coordinator combines these updates, typically by averaging them, into a new global model. The loop repeats until the global model converges.

This is local-update SGD wearing a privacy constraint. We already met the idea that workers can take several local steps between communication rounds to save network traffic; in the comm-efficient and local-SGD material of Section 10.7 it was an optimization to reduce the cost of the combine step. Federated learning reuses exactly that machinery but for a different reason: here the local steps are not merely a bandwidth saving, they are mandatory, because the only way a client can contribute is to compute on data the coordinator is never permitted to see. The averaging step that fuses the client models is the federated descendant of the all-reduce that opened this book, except that what is averaged is whole models trained in isolation rather than single-step gradients. Figure 14.1.1 shows the round in full.

2. Why This Matters Now Beginner

Federated learning is an old idea made urgent by three converging pressures. The first is law. Privacy regulation has tightened worldwide over the past decade, and the cost of mishandling personal data has risen from an engineering inconvenience to a board-level liability; "do not centralize the data" is increasingly a hard legal boundary rather than a courtesy. The second is where the data now lives. Smartphones, wearables, cars, and clinical instruments generate enormous volumes of data at the edge, far from any datacenter, and shipping all of it upstream is often impossible on the available uplink and undesirable for the people it describes. The third is the appetite for cross-institution collaboration: hospitals, banks, and manufacturers each hold a slice of a problem and would all benefit from a model trained on the union, if only there were a way to train on that union without forming it.

Federated learning is the way. It lets a model learn from data it is never allowed to see, which is exactly the situation that privacy law, edge data gravity, and competitive boundaries keep producing. This is why a technique first deployed at scale for phone keyboards around 2017 has since spread to medical imaging, finance, and industrial sensing, and why it now anchors a whole chapter of this book. The on-device and edge side of the story continues in Chapter 34, and a full clinical deployment, with all of its regulatory teeth, is the subject of the federated medical AI case study in Chapter 37.

3. A Model Trained Without Moving the Data Intermediate

It is fair to suspect that training in isolated pieces and averaging the results must give up something against training on the pooled data. For data that is identically distributed across clients, that suspicion is largely unfounded, and a small experiment shows why the basic recipe works before we spend later sections on the cases where it strains. We set up a linear-regression problem, split the examples across ten clients, and compare two models. The first is a central baseline trained on all the data at once, the model we would build if pooling were allowed. The second is a federated model produced by the protocol of Figure 14.1.1: each round, every client copies the global weights, runs several local gradient steps on its own shard, and returns only the updated weights, which the coordinator averages. This averaging of locally trained models is the algorithm known as federated averaging, or FedAvg, which Section 14.3 develops in full. No client's data is ever read by the coordinator or by any other client.

import numpy as np

rng = np.random.default_rng(0)
N, d, C = 60_000, 20, 10          # total examples, features, clients
X = rng.standard_normal((N, d))
w_true = rng.standard_normal(d)
y = X @ w_true + 0.1 * rng.standard_normal(N)

# Split the data across C clients. It NEVER leaves the client.
client_idx = np.array_split(rng.permutation(N), C)

def grad(w, Xs, ys):
    return (2.0 / len(ys)) * (Xs.T @ (Xs @ w - ys))

# --- Baseline: one machine sees ALL the data (the answer we want to match). ---
w_central = np.zeros(d)
lr, steps = 0.05, 300
for _ in range(steps):
    w_central -= lr * grad(w_central, X, y)

# --- Federated: coordinator broadcasts w, each client runs LOCAL updates on
#     its own private data, only the updated weight vector returns, the
#     coordinator AVERAGES them (FedAvg). No raw data is ever shared. ---
w_fed = np.zeros(d)
rounds, local_steps = 30, 10
for r in range(rounds):
    client_models = []
    for idx in client_idx:                       # each client, in isolation
        wk = w_fed.copy()                        # receive the global model
        Xs, ys = X[idx], y[idx]                  # private local shard
        for _ in range(local_steps):             # several LOCAL update steps
            wk -= lr * grad(wk, Xs, ys)
        client_models.append(wk)                 # send back ONLY the model
    w_fed = np.mean(client_models, axis=0)       # coordinator averages updates

def mse(w):
    return float(np.mean((X @ w - y) ** 2))

print("clients C                    :", C)
print("central MSE (all data)       :", f"{mse(w_central):.6f}")
print("federated MSE (no data move) :", f"{mse(w_fed):.6f}")
print("central vs federated  ||dw|| :", f"{np.linalg.norm(w_central - w_fed):.2e}")
print("federated vs true     ||dw|| :", f"{np.linalg.norm(w_fed - w_true):.2e}")

clients C                    : 10
central MSE (all data)       : 0.009953
federated MSE (no data move) : 0.009953
central vs federated  ||dw|| : 3.10e-05
federated vs true     ||dw|| : 1.47e-03

The two models are, for practical purposes, the same. The federated procedure reached a mean-squared error indistinguishable from the central baseline while every client trained alone on data the coordinator never read. Writing the global objective makes the target explicit. With $C$ clients, where client $k$ holds $n_k$ examples out of $N = \sum_k n_k$ and has local loss $F_k(w)$, federated training minimizes the size-weighted average

which is exactly the loss a central trainer would minimize on the pooled data, since pooling and then averaging the per-example losses gives the same sum. The clients optimize this shared objective collaboratively, each contributing the gradient signal from its own slice, and the coordinator's average is what stitches those contributions into progress on $F$. The clean agreement in Output 14.1.1 rests on a quiet assumption that the next sections dismantle: here the data was shuffled uniformly across clients, so every client saw the same distribution. When clients hold systematically different data, local models pull in different directions and the simple average drifts from the central answer, the central difficulty we take up in Section 4.

# pip install flwr ; the client wraps YOUR local training, data stays put
import flwr as fl

class Client(fl.client.NumPyClient):
    def fit(self, parameters, config):
        set_weights(model, parameters)           # receive global model
        train_locally(model, local_data)         # YOUR code, private data
        return get_weights(model), n_local, {}   # return ONLY weights

# coordinator side: FedAvg aggregation, no client data ever reaches here
fl.server.start_server(
    strategy=fl.server.strategy.FedAvg(),
    config=fl.server.ServerConfig(num_rounds=30),
)

4. What Federated Learning Makes Harder Intermediate

The clean result in Output 14.1.1 is the best case, and it is best because we engineered the data to be friendly. Bringing training to the data instead of the data to the training introduces three difficulties that datacenter training never had to face, and they are the reason the rest of this chapter exists. The first is non-IID data. In a datacenter we shuffle a central corpus so that every worker sees a representative sample; on real devices that is impossible. One phone's typing, one hospital's patient mix, and one bank's customer base are each systematically different from the others, so client objectives genuinely disagree and the average of locally trained models no longer lands where the central optimum sits. This is the dominant challenge of the field and gets its own treatment in Section 14.4.

The second difficulty is the clients themselves. Datacenter workers are uniform, always available, and connected by a fast interconnect. A federated population is the opposite: clients are heterogeneous and flaky, a phone participates only when it is idle, charging, and on unmetered Wi-Fi, and it may vanish mid-round. Uplink bandwidth is scarce and often metered, so the number of communication rounds and the size of each update become first-class constraints, which is why Section 14.5 is devoted to communication efficiency. The third difficulty is that sending model updates instead of raw data is necessary for privacy but not by itself sufficient, because updates can leak information about the data that produced them; closing that gap with secure aggregation and differential privacy is the subject of Section 14.6. That privacy thread runs straight through to the differentially private distributed training of Chapter 35.

We now have the inversion that defines this chapter (the data stays, the model travels), the basic protocol of broadcast, local training, and average, and a demonstration that it recovers the central answer when the data is friendly. The next section makes the first sharp distinction that organizes everything after it, between the cross-device regime of countless unreliable phones and the cross-silo regime of a few stable institutions, because almost every design choice in federated learning depends on which of those two worlds you are in. That split begins in Section 14.2.