Centralized, Decentralized, and Hybrid Architectures

"They asked who was in charge. I said everyone. They asked again, slower, and I gave the same answer, because that is the whole point of a gossip protocol."

A Peer With No Coordinator

Every system in this book partitions some essential activity across machines, but partitioning the work is only half the design. The other half is coordination: keeping the partitioned pieces consistent enough to produce a single coherent result. Coordination needs a source of truth, a place (or a procedure) that resolves disagreement and says what the current state is. The architectural choice is where that source of truth lives. At one extreme it lives in a single node that every other node defers to. At the other extreme there is no such node, and truth is whatever the peers collectively converge to. Between them sits a family of hierarchies that localize coordination to keep it cheap while preserving one global view. Naming these three patterns, and knowing the price of each, is the goal of this section.

1. The Centralized Pattern: One Coordinator Holds the Truth Beginner

In a centralized architecture, one machine is designated the coordinator and holds the authoritative copy of the shared state. Every other machine is a worker that reads the current state from the coordinator, does local work, and writes its contribution back. The coordinator serializes the contributions, applies them in a defined order, and answers the next read with the updated truth. The canonical instance in distributed machine learning is the parameter server: a node (or sharded set of nodes) holds the model weights, workers pull the current weights, compute gradients on their data shard, and push those gradients back, where the server applies them to produce the next version of the weights. The full push-pull mechanics, weight sharding, and the staleness that creeps in when pushes and pulls interleave are developed in Chapter 11.

The appeal of centralization is conceptual simplicity. There is exactly one place to look for the current state, one place to enforce an update order, and one place to add a new worker. Consistency is easy to reason about because the coordinator imposes a total order on updates by construction. Many systems beyond training share this shape: a scheduler that owns the cluster's placement decisions, a metadata server that owns the namespace of a distributed file system, a coordination service that owns the membership list. In each, a single logical authority trades scalability for a clean mental model.

2. The Decentralized Pattern: Peers Agree Without a Center Intermediate

A decentralized architecture removes the privileged node entirely. No machine holds the authoritative state; instead each peer holds its own copy, exchanges with a subset of other peers, and the group's agreed state is whatever those exchanges converge to. Two decentralized patterns dominate distributed training, and they sit at different points on the same axis. All-reduce over a ring or tree is decentralized in topology, every worker plays an identical role and there is no server, yet it still computes the exact global average each step by passing partial sums symmetrically around the ring; it is the workhorse of data-parallel training and is built up in detail in Chapter 4. Gossip averaging relaxes even the requirement of an exact global step: each peer mixes its value only with its immediate neighbors, and global agreement emerges gradually as information diffuses across the graph. Gossip and its federated relatives are the subject of Chapter 14.

The reward for giving up the center is that no single node can fail the whole system and no single link carries everyone's traffic; both taxes from Section 1 are dodged. The price is paid in convergence and visibility. Because no peer holds the global state, no peer has a global view: a peer knows only what has reached it through the graph, so a freshly changed value at one corner takes several rounds of exchange to influence the far corner. All-reduce keeps the per-step result exact but still pays a synchronization cost that grows with the ring. Gossip is cheaper per round but only approximate, converging toward the global average rather than hitting it in one step, and its convergence rate is governed by the connectivity of the communication graph, which is exactly why the control planes of Chapter 2 reach for consensus protocols when peers must agree on a single discrete value rather than a continuous average.

To make the contrast concrete, consider the simplest possible shared-state task: $K$ peers each hold a number, and they want to agree on the mean. A centralized solution sends all $K$ numbers to a coordinator, which averages them and broadcasts the result, exact in one round but funneled through one node. A gossip solution never gathers the numbers anywhere; each peer repeatedly replaces its value with a blend of its own and its ring neighbors' values, using a doubly stochastic mixing matrix $W$ so that the global sum is preserved while the values pull together. The update for the whole vector of peer values $g$ is $g \leftarrow W g$, and the deviation from the true mean shrinks geometrically at a rate set by the second-largest eigenvalue of $W$. The code below runs both and reports how the gossip peers close the gap to the global mean round by round.

import numpy as np

rng = np.random.default_rng(0)
K = 8                                  # peers, arranged in a ring
x = rng.uniform(-10, 10, size=K)       # each peer's local value
global_mean = x.mean()                 # the answer all peers should reach

# Centralized average: one coordinator gathers all values and broadcasts the mean.
central = np.full(K, x.mean())

# Decentralized gossip averaging on a ring: each round every peer replaces its
# value with a weighted blend of itself and its two ring neighbors. No peer ever
# sees the whole vector; information diffuses one hop per round.
g = x.copy()
W = 0.5 * np.eye(K)
for i in range(K):
    W[i, (i - 1) % K] += 0.25
    W[i, (i + 1) % K] += 0.25         # doubly-stochastic mixing matrix

print(f"global mean (target)        : {global_mean:.6f}")
print(f"central coordinator result  : {central[0]:.6f}")
print(f"{'round':>5}  {'max peer deviation from mean':>30}")
for r in range(1, 41):
    g = W @ g
    dev = np.max(np.abs(g - global_mean))
    if r in (1, 2, 5, 10, 20, 40):
        print(f"{r:>5}  {dev:>30.2e}")
print(f"gossip result (peer 0)      : {g[0]:.6f}")

global mean (target)        : 0.066019
central coordinator result  : 0.066019
round    max peer deviation from mean
    1                        8.22e+00
    2                        6.51e+00
    5                        3.58e+00
   10                        1.57e+00
   20                        3.21e-01
   40                        1.35e-02
gossip result (peer 0)      : 0.064894

The output tells the whole story of the trade-off in two columns. The centralized result is exact at once, because one node saw everything. The gossip result is only approximate after any finite number of rounds, and the deviation falls by a roughly constant factor each round, a geometric decay whose rate is the spectral gap of the ring. Add more peers or a sparser graph and that rate slows; wire the graph more densely and it speeds up, at the cost of more messages per round. This is the decentralized bargain in miniature: you trade the exactness and the single bottleneck of the center for many cheap local exchanges and a convergence you must wait for.

# Run with: torchrun --nproc_per_node=8 thisfile.py
import torch, torch.distributed as dist

dist.init_process_group("nccl")            # join the group of K peers, no server
val = torch.tensor([local_value], device="cuda")

# Decentralized AND exact: every peer ends with the global mean, no coordinator.
dist.all_reduce(val, op=dist.ReduceOp.SUM)
val /= dist.get_world_size()

# One gossip round instead: exchange only with ring neighbors (approximate, cheaper).
rank, world = dist.get_rank(), dist.get_world_size()
recv = torch.empty_like(val)
dist.send(val, dst=(rank + 1) % world)
dist.recv(recv, src=(rank - 1) % world)
val = 0.5 * val + 0.5 * recv               # mix with the neighbor that sent to us

3. The Hybrid Pattern: Hierarchies of Coordinators Intermediate

Pure centralization bottlenecks at one node; pure decentralization waits for slow global convergence. The hybrid pattern resolves the tension by nesting them. Machines are grouped, each group has a local coordinator that handles fast, frequent coordination within the group, and the local coordinators reconcile with a single global coordinator far less often. This is the right-hand panel of Figure 1.4.1, and it matches the physical reality of modern clusters: communication inside a server (across its GPUs over a fast interconnect) is cheap, communication across servers is dear, and communication across data centers is dearer still. A hierarchy places frequent exchange where bandwidth is abundant and rare exchange where it is scarce.

Hierarchical all-reduce is the standard concrete case. Workers on the same machine reduce their gradients locally over the fast intra-node link, one representative per machine then all-reduces across machines over the slower network, and the machine-level result is broadcast back down to the local workers. The expensive cross-machine collective runs once per step instead of once per worker, so a job with eight GPUs on each of sixteen machines pays for sixteen participants in the slow collective rather than one hundred twenty-eight. The same nesting appears in federated learning, where edge devices coordinate through a regional aggregator that in turn syncs with a global server, and in geo-distributed training, where each data center reduces internally before a thin cross-region exchange. Hierarchy is less an exotic third option than the practical default whenever the network is itself tiered.

4. Reading the Trade-offs Off the Spectrum Advanced

The three patterns are best held in mind as a single table of trade-offs rather than as a taxonomy to memorize. Table 1.4.1 lines them up against the four properties that decide most architectural arguments: where the source of truth lives, how the system tolerates a failure, where the communication bottleneck falls, and what kind of convergence the agreement step gives you. The running examples from this section, parameter server, all-reduce, and gossip, occupy the centralized, decentralized-exact, and decentralized-approximate rows; the hybrid pattern is the engineered compromise that picks a different row for different parts of the same job, exactly as the practical example did.

Two readings of the table are worth stating outright. First, no row dominates; the right choice depends on which resource is scarce and which failure you most fear, the same disciplined matching of remedy to ceiling urged in Section 1.1. If the model fits and the network inside a node is fast, hierarchical all-reduce is hard to beat; if peers join and leave constantly and an approximate average suffices, gossip is the robust answer; if the shared state is a giant sparse table of embeddings touched unevenly, a sharded parameter server earns its keep. Second, the patterns compose. A production training job often runs all-reduce for dense weights, a parameter server for sparse embeddings, and a consensus-backed coordinator for cluster membership, three points on the spectrum inside one system, which is why fluency with the whole spectrum, not allegiance to one pattern, is the skill this section builds.

We now have the third lens of this chapter in place. Section 1.1 split scale-out from scale-up, Section 1.2 mapped the six axes of distribution, Section 1.3 weighed scaling up against scaling out, and this section located the source of truth that any scaled-out system needs. The next dimension is temporal: whether the system processes a fixed batch, a flowing stream, one online example at a time, or an interactive request that must answer now. That distinction reshapes every architecture in this section, and it is the subject of Section 1.5.