Privacy and Secure Aggregation

"They promised the server would only ever see the sum. Then someone inverted my gradient and read my diary back to me, line by line."

A Client That Trusted the Aggregator

The previous section reduced the bytes each client sends, treating the update as a payload to compress. This section asks a sharper question about that same payload: what does it reveal? Federated learning's founding pitch is that raw data stays put and only model updates travel. That is a real improvement over shipping the data to a central store, but it is not, by itself, privacy. An update is a function of the data that produced it, and functions can be inverted. A curious or compromised server that receives one client's gradient can often run an optimization in the opposite direction, asking "what input would have produced this gradient?", and recover a recognizable copy of the client's training examples. The attack has a name, gradient inversion, and it works disturbingly well on the small batches typical of cross-device federated learning. So "we only send updates" must be hardened into "the server cannot tie any update to any client, and cannot tell whether any particular example was used at all." Those are two different guarantees, and they need two different tools.

1. Why Updates Leak Intermediate

Consider a single client computing the gradient of its loss on one labeled example $(x, y)$ with the current model $w$. The gradient $g = \nabla_w \ell(w; x, y)$ is a vector with as many entries as the model has parameters, and it depends on $x$ in a structured, often invertible way. For a fully connected layer, the gradient with respect to the weights is an outer product of the layer's input with the back-propagated error, and from that outer product the input can frequently be read off directly. An attacker who holds $g$ and knows the model architecture sets up an optimization over a dummy input $\hat{x}$ that minimizes $\lVert \nabla_w \ell(w; \hat{x}, \hat{y}) - g \rVert^2$, and for small batches the recovered $\hat{x}$ converges to a near-pixel-perfect copy of the original. A weaker but still serious attack, membership inference, does not reconstruct the example but decides whether a specific known example was in the training set, which is enough to leak that a particular patient's record contributed to a medical model.

The lesson is that an update is sensitive data, not a sanitized summary of it. Two properties would make federated aggregation safe. First, the server should never see an individual update in the clear, only the aggregate over many clients, so there is no single gradient to invert. Second, the aggregate itself should not reveal whether any one client's example was present, so that even the sum leaks nothing about an individual. Secure aggregation delivers the first property; differential privacy delivers the second. They compose, and serious deployments use both.

2. Secure Aggregation: The Server Sees Only the Sum Intermediate

Secure aggregation is a cryptographic protocol that lets a server compute $\sum_{k} u_k$, the sum of all client updates, without ever seeing any individual $u_k$. The core trick is elegant and needs no heavy machinery to understand: pairwise cancelling masks. Every pair of clients $(i, j)$ agrees on a shared random vector $m_{ij}$, derived from a secret they establish with a key exchange so that neither the server nor any third party knows it. Client $i$ adds $m_{ij}$ to its update and client $j$ subtracts the same $m_{ij}$. Each client sends its update plus the sum of all its pairwise masks. Individually, a masked message looks like uniform noise to the server. But when the server adds up all the masked messages, every mask $m_{ij}$ appears exactly twice, once with a plus sign from client $i$ and once with a minus sign from client $j$, so all masks cancel and the clean sum remains. Writing $u_k$ for client $k$'s update, what the server receives and then sums is

where $\operatorname{sgn}(i,j) = +1$ for the lower-indexed partner and $-1$ for the higher-indexed one. The masks vanish only because every pair contributes a matched $+m_{ij}$ and $-m_{ij}$; this is the same algebra of cancelling contributions that makes an all-reduce sum order-independent, the collective primitive built in Section 4.7. Secure aggregation is, in effect, an all-reduce whose intermediate values are cryptographically blinded. Figure 14.6.1 shows the flow: masked messages go up, masks cancel, only the sum comes down.

One subtlety makes secure aggregation more than a parlor trick: dropout tolerance. In cross-device federated learning many clients vanish mid-round (a phone locks, a network drops), and a client that fails to report leaves its masks uncancelled, which would corrupt the sum. Real protocols solve this with secret sharing: each client splits its mask seeds into shares distributed to others using a threshold scheme, so the server can reconstruct and remove the masks of clients that dropped, but only if at least a threshold number of clients survive the round. This is the privacy analogue of the dropout tolerance that Chapter 14's earlier sections build into the aggregation schedule: the masks cancel only when enough clients report, and below that threshold the server learns nothing, not even a partial sum. The runnable demo in Section 4 implements the masking core; production handles the secret-sharing recovery for you.

3. Differential Privacy: Bounding What One Client Reveals Advanced

Secure aggregation hides individual updates, but the sum it reveals can still leak information about a single client if that client's contribution dominates. Differential privacy (DP) closes this by a formal guarantee: a randomized mechanism $\mathcal{M}$ is $(\varepsilon, \delta)$-differentially private if, for any two datasets $D$ and $D'$ that differ by one client's data and any set of outcomes $S$,

Read plainly: adding or removing any single client changes the distribution of outputs by at most a multiplicative factor $e^{\varepsilon}$ (plus a small slack $\delta$). A small $\varepsilon$ means strong privacy, because the output barely depends on whether your data was included, so an adversary cannot confidently infer your membership. The standard recipe to achieve this for federated averaging has two steps. First, clip each client's update to a bounded $L_2$ norm $C$, so no client can move the aggregate by more than a known amount; this bounds the mechanism's sensitivity. Second, add Gaussian noise with standard deviation proportional to that bound, $\sigma = z\, C / K$ on the average of $K$ clients, where the noise multiplier $z$ sets the strength. The Gaussian mechanism then satisfies $(\varepsilon, \delta)$-DP with

so a larger noise multiplier $z$ directly buys a smaller $\varepsilon$, that is, stronger privacy. The price is accuracy: the injected noise perturbs the averaged update, and the model converges to a slightly worse solution. This is the privacy-utility trade-off, and it is not optional folklore but a quantity you tune. We add only a forward pointer here, because the full accountant machinery that tracks how $\varepsilon$ accumulates over many training rounds, and how DP behaves across a distributed system, belongs to Chapter 35; the clinical setting where these guarantees become a regulatory requirement is the subject of the federated medical AI case study in Chapter 37.

4. Building It: Masked Aggregation and the Privacy-Utility Trade-off Intermediate

The code below implements both defenses on small vectors so the mechanism is visible. The secure-aggregation half builds pairwise masks from shared per-pair seeds, has each client send only its blinded message, and shows the server recovering the exact sum while a single masked message reveals nothing recognizable about that client's true update. The differential-privacy half clips each update to a norm bound and adds Gaussian noise to the average, sweeping the noise multiplier $z$ to measure how the error against the clean (non-private) average grows as privacy strengthens.

import numpy as np

# ---- Setup: K clients, each holds a private d-dimensional model update ----
rng = np.random.default_rng(7)
K, d = 5, 4
updates = rng.standard_normal((K, d))          # the secret each client wants to hide
true_sum = updates.sum(axis=0)                  # what the server is allowed to learn

# ---- Secure aggregation: pairwise masks that cancel in the sum ----
# For each unordered pair (i, j) both clients derive the SAME random mask from a
# shared seed (in practice via a Diffie-Hellman key exchange). Client i adds it,
# client j subtracts it. Summed over all clients, every mask appears once with a
# + and once with a -, so the masks vanish and only the true sum survives.
masks = np.zeros((K, d))
for i in range(K):
    for j in range(i + 1, K):
        seed = 1000 * (i + 1) + (j + 1)         # shared pairwise seed
        m = np.random.default_rng(seed).standard_normal(d)
        masks[i] += m                           # client i adds m_{ij}
        masks[j] -= m                           # client j subtracts m_{ij}

masked = updates + masks                        # what each client actually sends
recovered = masked.sum(axis=0)                  # server sums the masked messages

print("=== Secure aggregation (masks cancel in the sum) ===")
print("a single masked message (server sees this):", np.round(masked[0], 3))
print("that client's true update (server never sees):", np.round(updates[0], 3))
print("recovered sum    :", np.round(recovered, 6))
print("true sum         :", np.round(true_sum, 6))
print("max abs sum error:", f"{np.max(np.abs(recovered - true_sum)):.2e}")

# ---- Differential privacy: clip per-client, then add Gaussian noise ----
# Bound each client's contribution by clipping its L2 norm to C, then add
# Gaussian noise with std sigma = z * C / K to the AVERAGED update. Larger z
# means stronger privacy (smaller epsilon) but a noisier, less accurate result.
def dp_average(updates, C, z, key):
    clipped = np.array([u * min(1.0, C / np.linalg.norm(u)) for u in updates])
    avg = clipped.mean(axis=0)
    noise = np.random.default_rng(key).normal(0.0, z * C / K, size=d)
    return avg + noise

clean_avg = updates.mean(axis=0)
print("\n=== Differential privacy (clip + Gaussian noise) ===")
print(f"{'noise mult z':>12} | {'avg L2 error vs clean':>22}")
for z in (0.0, 0.5, 1.0, 4.0):
    errs = [np.linalg.norm(dp_average(updates, C=2.0, z=z, key=s) - clean_avg)
            for s in range(2000)]
    print(f"{z:>12.1f} | {np.mean(errs):>22.4f}")

=== Secure aggregation (masks cancel in the sum) ===
a single masked message (server sees this): [-1.611 -1.698 -0.706 -1.313]
that client's true update (server never sees): [ 0.001  0.299 -0.274 -0.891]
recovered sum    : [-2.184447 -2.70146  -1.654627  0.212276]
true sum         : [-2.184447 -2.70146  -1.654627  0.212276]
max abs sum error: 4.44e-16

=== Differential privacy (clip + Gaussian noise) ===
noise mult z |  avg L2 error vs clean
         0.0 |                 0.1401
         0.5 |                 0.3993
         1.0 |                 0.7633
         4.0 |                 3.0048

Two facts in the output deserve emphasis. The masked message $[-1.611, -1.698, \dots]$ looks nothing like the client's true update $[0.001, 0.299, \dots]$, yet summing five such blinded messages reproduces the true sum exactly: that is the whole value proposition of secure aggregation, perfect aggregate fidelity with zero individual exposure. In the DP half, the error is not noise added for its own sake; it is the precise, tunable price of the $(\varepsilon, \delta)$ guarantee from Section 3. A deployment chooses where on that curve to sit, and the small residual error at $z=0$ is a reminder that clipping alone, before any noise, already perturbs the result.

# pip install opacus
from opacus import PrivacyEngine

privacy_engine = PrivacyEngine()
model, optimizer, data_loader = privacy_engine.make_private(
    module=model, optimizer=optimizer, data_loader=data_loader,
    noise_multiplier=1.0,        # the z from Code 14.6.1
    max_grad_norm=1.0,           # the clipping bound C
)
# train as usual; Opacus clips per-sample grads, adds noise, and tracks epsilon
epsilon = privacy_engine.get_epsilon(delta=1e-5)

5. The Trade-off You Must Price Intermediate

Privacy is never free, and the two defenses cost differently. Secure aggregation is, pleasingly, lossless on accuracy: the recovered sum is exact, as Output 14.6.1 confirms, so its cost is purely in communication and protocol complexity (extra key-exchange rounds, secret-sharing for dropout, and bandwidth for masks the size of the update). Differential privacy, by contrast, is lossy by design: the noise that buys the $(\varepsilon, \delta)$ guarantee also degrades the model, and a smaller $\varepsilon$ means a noisier, weaker model. The engineering question is therefore not "should we add privacy?" but "what is the smallest $\varepsilon$ we can promise while keeping accuracy acceptable, and have we removed every individual update from the server's view first?" In a well-built system the answer combines both: secure aggregation makes the server's view an aggregate, and a modest amount of differential-privacy noise on that aggregate bounds residual leakage, often at a far smaller accuracy cost than DP applied to individual updates would incur, because noise added once to a sum of thousands is proportionally tiny.

With privacy in place, the federated story has its safety guarantee: data stays local, no individual update is ever exposed, and the formal $(\varepsilon, \delta)$ bound limits what even the aggregate can betray. The next question is one of accuracy rather than secrecy. Clients hold different data distributions, and a single global model may serve none of them well; Section 14.7 turns to personalized federated learning, where each client adapts the shared model to its own data while still benefiting from the crowd.