The Distributed AI Design Space

"They asked me to design a distributed system. I asked them which ceiling was on fire. The room went quiet, and then we finally had a real conversation."

A Coordinator That Learned to Ask First

A practitioner who has read this far can recite the parts of a distributed AI system. The harder skill, the one that separates a working design from an expensive misadventure, is sequencing the decisions so that each one constrains the next. You do not pick a topology and then discover which ceiling binds; you find the binding ceiling first, because it dictates which axis you must distribute, which in turn narrows the sensible topologies, which sets the processing mode, which exposes the metric you are really optimizing, which finally fixes the communication and failure tax you have to pay. Reverse that order and you get systems that are distributed along the wrong axis, optimized for the wrong metric, and surprised by a tax nobody budgeted for. The design space is large, but the path through it is short and almost always the same.

1. The Checklist, Stage by Stage Beginner

The checklist is six questions asked in the order of Figure 1.8.1. Each draws directly on one earlier section, so you can think of this stage as a compression of the whole chapter into a procedure. Read each question, then commit to one answer before moving on; the discipline is in the sequencing, not in the cleverness of any single choice.

Stage 1, which ceiling binds. Before anything else, name the single resource that ran out. Section 1.1 gave you three candidates: the data is too big for one machine's memory or disk, the model is too big for one accelerator's memory, or the request volume exceeds one server's throughput. Usually one of the three dominates; occasionally two do, and a foundation model hits all three at once. If none binds, the correct design is a single machine, and the rest of this checklist is moot. Distribution is forced by a ceiling, never chosen for elegance.

Stage 2, which axis to distribute. The binding ceiling maps almost mechanically onto one of the six axes from Section 1.2. A data ceiling sends you to distribute data (Part II). A throughput ceiling on training sends you to distribute training (Part III); a throughput ceiling on serving sends you to distribute inference (Part V). A model ceiling sends you to distribute the model (Part IV). A system whose difficulty is keeping many machines efficient and recoverable leans on cluster coordination (Parts I and VII), and a system of many reasoning entities distributes intelligence (Part VI). Distribute along the axis the ceiling points to; distributing along any other axis adds cost without relief.

Stage 3, which topology. With the axis chosen, Section 1.4 narrows the architecture. A centralized topology (one coordinator, many workers) is simplest and is the right default when the coordinator is not itself a bottleneck and a single trust domain is acceptable. A decentralized topology (peers, no single coordinator) is forced when no party may hold all the data or when a single point of failure is unacceptable, as in federated learning (Chapter 14). A hybrid topology, centralized within a region and decentralized across regions, is what most large systems actually become. Pick the simplest topology the axis and the trust boundaries allow.

Stage 4, which processing mode. Section 1.5 asks how work arrives and how fresh the answer must be. Batch processes a bounded dataset for maximum throughput with relaxed latency. Streaming processes an unbounded feed with bounded latency. Online updates the model itself from each arriving example. Interactive answers a user within a human-perceptible budget. The mode is largely dictated by the product, not chosen freely, but stating it explicitly prevents the classic error of building a batch system for an interactive requirement.

Stage 5, which dominant metric. Section 1.6 named four objectives in tension: throughput, latency, cost, and reliability. You cannot maximize all four; the design must declare which one it optimizes and which it merely satisfies as a constraint. A training pipeline optimizes throughput per dollar; an interactive assistant optimizes tail latency; a payments system optimizes reliability and treats latency as a hard constraint. Naming the dominant metric tells you which trade-offs are allowed and which are forbidden.

Stage 6, the tax to budget. Every distribution choice incurs two recurring costs that Section 1.1 and Section 1.6 introduced and that the rest of the book quantifies: the communication tax (bytes moved between machines per unit of work) and the failure tax (the probability that, at any moment, something in the system is broken, and the redundancy you must run to survive it). Estimate both before you commit. If the communication tax per step exceeds the work it enables, more machines make the system slower, and you must loop back to stage one with fewer workers. We make these estimates rigorous with the alpha-beta cost model of Chapter 3.

2. The Design Space as Six Axes You Can Score Intermediate

The checklist produces categorical answers, but it helps to see the whole design as a single shape. Assign each of the six distribution axes a pressure in $[0,1]$, the degree to which that axis is forced for the system at hand, and the design becomes a point in a six-dimensional space that you can read at a glance. Two systems with different products but the same pressure profile will share most of their engineering; two systems that look superficially alike but differ in which axis dominates will diverge sharply in their architecture. Table 1.8.1 lays out the full checklist as a reusable worksheet, one row per stage, with the section that owns each question and the part of the book that develops the answer.

The two taxes in the final row are the only entries that feed back. A topology or axis choice that looks correct in isolation can produce a communication cost so large that it becomes the new binding ceiling, which is the dashed back-edge in Figure 1.8.1. This is why the checklist is a flow with one loop rather than a straight line: you may have to walk it twice, the second time with a smaller worker count or a different collective, before the design is stable. Chapter 3 gives you the arithmetic to detect this in advance instead of in production.

3. The Checklist Applied, End to End Intermediate

A checklist earns its place only when you watch it decide a real case. Take a concrete system: a real-time card-fraud scoring service that must score every transaction before it is authorized. The model is a gradient-boosted tree ensemble that fits comfortably in one machine's memory; the corpus of historical transactions is large but not web-scale; the binding pressure is the request rate, tens of thousands of authorizations per second, each under a hard tail-latency budget. We walk the six stages, and the small program below scores the axes and computes the tax so the decision rests on numbers rather than intuition.

# Apply the design-space checklist to a concrete system as a small scoring model.
# Worked example: a real-time fraud-scoring service.
# Each of the six axes gets a "bind pressure" in [0,1]; we pick the dominant ones.

axes = {
    "data":         0.30,  # transaction history large but not web-scale
    "training":     0.40,  # nightly retrain, moderate
    "model":        0.10,  # gradient-boosted trees, fits one node easily
    "inference":    0.95,  # 50k requests/sec, hard p99 latency SLA -> binds
    "coordination": 0.55,  # fleet must be scheduled, recovered
    "intelligence": 0.05,  # single decision, no multi-agent reasoning
}

# Ceiling test: which of data / model / throughput binds?
ceilings = {"data": 0.30, "model": 0.10, "throughput": 0.95}
binding = max(ceilings, key=ceilings.get)

# Communication + failure tax estimate (toy alpha-beta intuition):
# per-request fan-out of F feature lookups, each a network hop of latency alpha.
alpha_ms = 0.4         # one network hop, microseconds-to-ms regime
F = 6                  # feature-store lookups per scoring request
comm_tax_ms = F * alpha_ms
slo_ms = 20.0
budget_left = slo_ms - comm_tax_ms

print("dominant ceiling      :", binding, f"(pressure {ceilings[binding]:.2f})")
top = sorted(axes.items(), key=lambda kv: -kv[1])[:2]
print("axes to distribute    :", ", ".join(f"{k}" for k, _ in top))
print("comm tax per request  :", f"{comm_tax_ms:.1f} ms of {slo_ms:.0f} ms SLO")
print("compute budget left   :", f"{budget_left:.1f} ms")
print("verdict               :", "scale OUT inference, replicate; keep model on one node"
      if binding == "throughput" else "reconsider")

dominant ceiling      : throughput (pressure 0.95)
axes to distribute    : inference, coordination
comm tax per request  : 2.4 ms of 20 ms SLO
compute budget left   : 17.6 ms
verdict               : scale OUT inference, replicate; keep model on one node

Now read the verdict back through the six stages in prose. Stage 1: throughput binds, at pressure $0.95$, far above data and model. Stage 2: the throughput ceiling is on serving, so the axis is distribute inference, with cluster coordination as the secondary axis because a large serving fleet must be scheduled and recovered. Stage 3: the topology is centralized, a load balancer in front of stateless replicas, since there is one trust domain and the balancer is not the bottleneck. Stage 4: the mode is interactive, one answer per transaction inside a $20$ ms budget. Stage 5: the dominant metric is tail latency (the p99 must hold), with reliability as a hard constraint and cost a softer one. Stage 6: the communication tax is the $2.4$ ms of feature-store lookups, well inside budget, and the failure tax is the redundancy needed to keep the p99 intact when a replica dies, which is why coordination scored as high as it did. The model never needed splitting, the data never needed a distributed training run for this requirement, and a team that had reached for model parallelism or a decentralized topology would have solved a ceiling that was never binding. That is the entire value of walking the stages in order.

# Distributed AI design checklist -> the tool that handles each axis.
PLAYBOOK = {
    "distribute data":       "PySpark / Ray Data   # shard + shuffle a >1-node corpus",
    "distribute training":   "torch DDP            # gradient all-reduce across workers",
    "distribute the model":  "FSDP / DeepSpeed     # shard params + optimizer state",
    "distribute inference":  "vLLM / Ray Serve     # replicate, batch, autoscale",
    "coordinate the cluster":"Kubernetes / Slurm   # schedule, restart, place by topology",
    "distribute intelligence":"a multi-agent runtime # orchestrate reasoning agents",
}
# The fraud example's verdict, "distribute inference + coordinate", reads off as:
print(PLAYBOOK["distribute inference"])
print(PLAYBOOK["coordinate the cluster"])

4. The Tax You Always Pay, and the Frontier That Lowers It Intermediate

Stage 6 is the stage practitioners most often skip, and it is the one the rest of the book spends the most pages making rigorous. The communication tax is not a nuisance term; for many designs it is the quantity that decides how many machines actually help. Recall the gradient identity of Section 1.1: the combine step that made data parallelism exact was free only because the data lived in one process. Across a real network, that same all-reduce moves the entire gradient between machines every step, and its cost grows with the model size and the worker count while the useful compute per worker shrinks. The honest worker count is the one where the marginal communication tax has not yet swallowed the marginal compute, and finding it is the measurement Chapter 3 formalizes and Chapter 4 turns into the cost of specific collectives.

The failure tax is the second half of stage 6 and follows the same logic. One machine either works or it does not; a thousand machines are, at any instant, a system in which something is probably broken. A design that ignores this passes its first demo and fails its first week in production. Budgeting the failure tax means deciding, before launch, how much redundancy and how much re-execution you will run to keep the dominant metric intact when a node dies, a discipline that runs from MapReduce re-execution (Chapter 6) to elastic training (Chapter 18).

5. Chapter Summary and the Capstone Handoff Beginner

This section closes Chapter 1, so it is worth stating the spine the whole chapter built. We began with a single calculation (Section 1.1) showing that the most important form of distribution, the data-parallel gradient, is exact rather than approximate, which is why scale-out is worth its complexity at all. We named the six axes along which an AI system can be distributed (Section 1.2), separated scaling out from scaling up and committed the book to the former (Section 1.3), surveyed the centralized, decentralized, and hybrid topologies those axes live in (Section 1.4), the batch, streaming, online, and interactive modes they run in (Section 1.5), and the throughput, latency, cost, and reliability metrics they are judged by (Section 1.6). Four real systems showed the axes stacking in practice (Section 1.7). This final section folded all of it into one procedure: find the binding ceiling, let it choose the axis, let the axis narrow the topology, let the product fix the mode, declare the dominant metric, and budget the communication and failure tax, looping once if the tax itself becomes the ceiling.

The capstone (Chapter 41) is where this checklist stops being a teaching device and becomes your deliverable. Its sections walk the same six stages under different names: choose a distributed AI problem, define the distribution axis, build a single-machine baseline, design the distributed version, select tools, and analyze cost and performance. The worksheet in Table 1.8.1 is, almost line for line, the rubric that capstone is graded against. The Project Ideas at the close of this chapter (see the chapter overview page) are starting points sized so that one of them, carried through this checklist, becomes a capstone.

That closes Chapter 1. You arrived able to write "scale-out AI" on a whiteboard; you leave with a procedure that turns any AI system you encounter into six answered questions and a budgeted tax. The next chapter steps under the hood of the machines this checklist coordinates. Before you can budget a communication or failure tax with confidence, you need the vocabulary of distributed systems: what a process, a message, and a clock actually guarantee, why "the network is reliable" is a falsehood every distributed AI system is built to survive, and how consistency, coordination, and recovery are made precise. Chapter 2 builds exactly that foundation, and every later part stands on it.