Distributed System Patterns for AI

"After my third distributed system I stopped learning new architectures. I started recognizing old ones wearing new framework names."

A Coordinator That Has Seen This Shape Before

A pattern is not a framework and not an algorithm; it is the shape of the data flow and the coordination, independent of the library that implements it. The same parameter-server pattern underlies a 2014 recommendation system and a 2025 sparse-embedding store, even though the code shares not a single line. Recognizing the shape is what lets you carry intuition from one system to the next: once you know that all-reduce makes every worker hold the same averaged result in lockstep, you know its synchronization cost and its straggler sensitivity before you read a word of the framework's documentation. Each of the six patterns below names one such shape, states where its state lives and how it synchronizes (the concerns of Section 2.2 and Section 2.3), how it handles a failed node (Section 2.4), and the workload it was born to serve. The forward links are deliberate: this section is a table of contents for the engine room of the book.

1. Six Patterns, One Vocabulary Beginner

We catalog six patterns. They are not exhaustive, but together they account for the large majority of the distributed AI systems you will build or read about, and every later chapter is, in structural terms, a deep treatment of one of them. Read each pattern as a triple: the shape of its state, its synchronization discipline, and the one-line rule for when it is the right choice.

Pattern 1: parameter server (central state, push and pull). One logical store, itself sharded across nodes, holds the authoritative model parameters. Workers pull the parameters they need, compute an update on local data, and push the update back; the server applies it. State is central and mutable; synchronization can be tight or deliberately loosened into bounded staleness (Section 2.5); a failed server shard is recovered from a replica or checkpoint, a failed worker simply stops contributing. When to use: the parameter set is huge and sparse, so each worker touches only a slice of it, as with billion-row embedding tables. This pattern is the subject of Chapter 11.

Pattern 2: all-reduce and collective SGD (no central state, symmetric peers). There is no server. Every worker holds a full replica of the model, computes a gradient on its data shard, and the peers combine their gradients with an all-reduce so that all of them end the step holding the identical averaged result. State is replicated, not central; synchronization is tight and symmetric; a failed worker stalls the collective until the group is reformed. When to use: the model fits on one device and the binding ceiling is training throughput, so you replicate the model and split the data. The collective itself is built in Chapter 4, and it becomes a training loop in Chapter 15.

Pattern 3: MapReduce and scatter-gather (data-parallel, stateless tasks). A coordinator scatters a bounded dataset into independent tasks, each processes its shard with no inter-task communication, and the results are gathered (and optionally shuffled and reduced) into the output. State lives in the data, not in the workers; there is no per-step synchronization, only a barrier between phases; a failed task is simply re-executed because it is deterministic and side-effect-free. When to use: an embarrassingly parallel pass over a dataset too big for one machine, such as deduplicating or tokenizing a web crawl. This pattern, and the re-execution fault tolerance that makes it robust, is Chapter 6.

Pattern 4: actor-learner (decoupled generation and training). Many actor processes interact with environments to generate experience, and one or a few learner processes consume that experience to update a policy, which is then shipped back to the actors. State is split: transient experience on the actors, authoritative policy on the learner; synchronization is loose and asynchronous by design, so a slow actor never blocks the learner; a failed actor is just restarted and its lost experience is regenerated. When to use: reinforcement learning, where collecting experience is the throughput bottleneck and must scale independently of the gradient step. This is the spine of Chapter 20.

Pattern 5: sharded and replicated serving fleet (stateless replicas behind a balancer). The model is replicated across many serving nodes (and, for large models, each replica is itself sharded across devices); a load balancer routes each request to a healthy replica. State is the (read-only) model weights plus per-request transient state; there is no cross-replica synchronization on the hot path; a failed replica is removed from rotation and its traffic shifts to the survivors. When to use: the binding ceiling is inference throughput or tail latency under a strict budget. This is Chapter 23.

Pattern 6: hierarchical and decentralized coordination (peers, gossip, and aggregation trees). No single node holds all the state or coordinates all the others. Workers organize into a tree or a peer graph and exchange partial results with neighbors, aggregating up a hierarchy or mixing through gossip, so that information spreads without any central bottleneck or single trust domain. State is partitioned and locally owned; synchronization is partial and often asynchronous; the structure tolerates node loss because no node is irreplaceable. When to use: when no party may hold all the data (privacy), when nodes are geographically far apart, or when a central coordinator would be a bottleneck or a single point of failure. Federated and decentralized learning, in Chapter 14, is the canonical instance.

2. The Patterns Side by Side Beginner

Listing the patterns one by one hides what is clearest in a single view: they differ along exactly the axes Chapter 2 spent its sections building. Table 2.9.1 lays all six against where their state lives (Section 2.3), their synchronization model (Section 2.2 and Section 2.5), how they handle a failed node (Section 2.4), and the workload each fits best. Read down any column to compare the patterns on one concern; read across any row to get a pattern's full profile at a glance.

Two structural contrasts in Table 2.9.1 are worth naming because they recur throughout the book. The first is central versus symmetric state: the parameter server keeps one authoritative copy that workers read and write, while all-reduce keeps a full copy on every peer and reconciles them each step. The same training job can be expressed either way, and the choice is a communication-cost trade-off that Chapter 4 makes precise. The second is synchronous versus asynchronous coupling: all-reduce is lockstep, so one straggler (Section 2.7) stalls everyone, whereas the actor-learner pattern deliberately decouples generation from training so that a slow actor costs only its own share of experience. Figure 2.9.1 draws the three most contrasting shapes so you can see the data flow rather than read it.

3. Matching a Workload to Its Pattern Intermediate

The practical skill is not memorizing the six patterns but matching a workload to the one that fits. A workload exerts pressure along a few concerns: how much one authoritative copy of state must be shared, how synchronous the steps must be, how much the work fans out into independent shards, and how much the binding pressure is request throughput. Each pattern is good at a particular profile of those concerns. The program below scores five representative AI workloads against the six patterns using a simple cosine match between each workload's pressure vector and each pattern's strength vector, and reports the best fit. It is a teaching toy, not a production planner, but it makes the matching logic explicit and reproducible.

# Six distributed AI patterns scored against one workload's pressure profile.
# Each pattern is rated on how well it fits a given pressure vector, then ranked.
import numpy as np

# A workload's pressure on four concerns, each in [0, 1].
#   central_state : how much one authoritative copy of state must be shared
#   sync_tight    : how synchronous the steps must be (1 = lockstep, 0 = loose)
#   fan_scatter   : how much the work fans out into independent shards
#   serve_qps     : how much the binding pressure is request throughput
workloads = {
    "data-parallel LLM training": dict(central_state=0.5, sync_tight=0.9, fan_scatter=0.4, serve_qps=0.1),
    "huge sparse recsys embeddings": dict(central_state=0.95, sync_tight=0.3, fan_scatter=0.6, serve_qps=0.3),
    "batch corpus dedup":          dict(central_state=0.1, sync_tight=0.1, fan_scatter=0.95, serve_qps=0.0),
    "online RL from a simulator":  dict(central_state=0.6, sync_tight=0.2, fan_scatter=0.8, serve_qps=0.2),
    "low-latency model serving":   dict(central_state=0.1, sync_tight=0.1, fan_scatter=0.3, serve_qps=0.95),
}

# Each pattern is a weight vector over the same four concerns: what it is good at.
patterns = {
    "parameter server":          np.array([0.95, 0.30, 0.50, 0.30]),
    "all-reduce / collective":   np.array([0.40, 0.95, 0.50, 0.10]),
    "MapReduce / scatter-gather":np.array([0.10, 0.10, 0.95, 0.05]),
    "actor-learner":             np.array([0.55, 0.20, 0.90, 0.25]),
    "sharded/replicated serving":np.array([0.15, 0.10, 0.40, 0.95]),
    "hierarchical/decentralized":np.array([0.30, 0.25, 0.60, 0.40]),
}
keys = ["central_state", "sync_tight", "fan_scatter", "serve_qps"]

for wname, w in workloads.items():
    v = np.array([w[k] for k in keys])
    scores = {p: float(np.dot(v, pw) / (np.linalg.norm(v) * np.linalg.norm(pw)))
              for p, pw in patterns.items()}
    best = max(scores, key=scores.get)
    print(f"{wname:30s} -> {best:28s} (fit {scores[best]:.3f})")

data-parallel LLM training     -> all-reduce / collective      (fit 0.992)
huge sparse recsys embeddings  -> parameter server             (fit 0.997)
batch corpus dedup             -> MapReduce / scatter-gather   (fit 0.999)
online RL from a simulator     -> actor-learner                (fit 0.995)
low-latency model serving      -> sharded/replicated serving   (fit 0.995)

The clean separation in Output 2.9.1 is a feature of the chosen examples, not a claim that every system maps to exactly one pattern. Real systems compose patterns, and the most demanding ones use several at once. Training a large language model uses all-reduce for the data-parallel dimension, a sharded (parameter-server-like) layout for the optimizer state, and a MapReduce-style pass to prepare the corpus, then serves the result from a sharded replica fleet. The point of the catalog is not to force a system into one box but to give you the names for the boxes, so that you can describe a complex system as a composition of patterns whose individual costs and failure modes you already understand.

# Distributed AI pattern -> the production library that implements it.
PATTERN_TOOL = {
    "parameter server":          "TorchRec / a PS-backed embedding store  # shard huge sparse tables",
    "all-reduce / collective":   "torch DDP over NCCL                      # gradient all-reduce",
    "MapReduce / scatter-gather":"PySpark / Ray Data                       # scatter, shuffle, reduce",
    "actor-learner":             "Ray RLlib                                # actors generate, learner trains",
    "sharded/replicated serving":"vLLM / Ray Serve                         # replicate, batch, autoscale",
    "hierarchical/decentralized":"Flower / a federated-learning runtime    # aggregate without central data",
}
chosen = "all-reduce / collective"          # e.g. the verdict for data-parallel training
print(PATTERN_TOOL[chosen])

4. Why These Patterns Persist Intermediate

It is fair to ask why a handful of patterns, several of them more than a decade old, keep reappearing under new framework names. The reason is that each pattern is the natural answer to a specific combination of the constraints Chapter 2 made precise, and those constraints do not change when the hardware does. The parameter server exists because some state is too large and too sparse for every worker to hold a copy; that is a property of the data, not of any GPU generation. All-reduce exists because, when the model does fit, replicating it and reconciling gradients is provably exact (the gradient identity of Section 1.1, seen again as a BSP barrier in Section 2.2) and avoids a central bottleneck. MapReduce persists because deterministic, side-effect-free tasks are the cheapest possible thing to make fault tolerant, and a web-scale data pass needs exactly that. The patterns are stable because the trade-offs they resolve are stable.

What does change is the engineering around each pattern, and that is where the current frontier lives. The patterns are fixed points; the research is in lowering their costs and blurring their boundaries.

5. Chapter Summary Beginner

This section closes Chapter 2, so it is worth restating the arc the whole chapter built. We started from the actors of a distributed system, processes, nodes, workers, and coordinators (Section 2.1), and the ways they talk and stay in step (Section 2.2). We saw how state is spread by partitioning, sharding, and replication (Section 2.3), and why that spreading makes failure a design problem rather than an afterthought, met with checkpointing, re-execution, and recovery (Section 2.4). We made precise what "consistent" can mean when a parameter store is read and written by many workers, from staleness bounds to the CAP trade-off (Section 2.5), and separated the control plane, where consensus and leader election rule (Section 2.6), from the data plane, where collectives do. We named the stragglers and bottlenecks that decide real performance (Section 2.7) and the locality, of data and of compute, that you arrange to avoid moving bytes you did not have to (Section 2.8). This final section folded all of those concerns into six reusable patterns and showed that every distributed AI system you will meet is one of them, or a composition of a few.

With the concepts of Chapter 2 compressed into patterns, the next question is quantitative: given a pattern, exactly how well does it scale, and where does adding machines stop helping? That is the subject of Chapter 3, which gives you Amdahl's and Gustafson's laws, the roofline model, and the alpha-beta cost of communication, the arithmetic that turns "this pattern should scale" into a number you can defend.