Defining the Distribution Axis

"They added eight more of me before anyone asked what was actually slow. I am very fast now at the thing that was never the problem."

A Bottleneck, Asking to Be Named Before It Is Cut

The previous section framed the capstone as a problem statement: a workload that one machine can no longer serve, and a goal stated in measurable terms. This section makes the first irreversible design decision. Of the many things a system does, you will distribute one of them, and the rest of the project, the baseline you build in Section 41.3 and the architecture you commit to in Section 41.4, follows from that choice. Choosing well is not a matter of taste or of which technique you find most interesting. It is a matter of finding the one place where the single machine breaks and pointing your effort precisely there. Distribution is forced by a ceiling, never chosen for elegance, and the discipline of this section is refusing to commit until you can name the ceiling.

1. The Six Axes, Recalled as a Menu of Remedies Beginner

Chapter 1 introduced six axes along which an AI system can be distributed, and Section 1.2 walked each with a worked example. Here we recall them not as a taxonomy but as a menu of remedies, because every axis answers a different question and therefore cures a different ailment. Distributing data spreads the dataset and its processing across machines, so it cures a system drowning in bytes it cannot read or store. Distributing training spreads the gradient computation across workers, so it cures a system whose compute is the limit while the model still fits on one device. Distributing the model splits the parameters and activations across devices, so it cures a system whose model is too large to hold at all. Distributing inference replicates and partitions the serving path, so it cures a system that cannot answer requests fast enough. Coordinating the cluster and distributing intelligence handle the scheduling, recovery, and multi-agent reasoning that hold the whole thing together once several machines are involved.

The trap is treating these as interchangeable knobs. They are not. Each axis removes exactly one kind of ceiling, and applying the wrong one leaves the real ceiling untouched while adding the communication tax that Chapter 3 taught you to price. Table 41.2.1 lays the menu out as a lookup: the symptom you measure, the axis that cures it, the part of the book that develops the technique, and the production tool you would reach for. The columns after the symptom are useless until you have measured the symptom, which is why the rest of this section is about measurement.

2. Diagnosing the Binding Bottleneck Intermediate

To choose an axis you must answer one question about the single-machine baseline: which resource does it exhaust first? There are five candidates, and they map cleanly onto the menu. A workload is data I/O bound when it spends its time reading or writing bytes, the processor idling while the disk or network catches up. It is compute bound when the arithmetic units are saturated and feeding them faster would not help. It is memory bound in two distinct senses worth separating: memory-bandwidth bound when the cores starve waiting for data to stream from RAM, and memory-capacity bound when the working set simply does not fit and the program cannot run at all without paging or crashing. It is latency bound when the clock that matters is the time to first response, not total throughput. And it is coordination bound when the time goes to synchronization, queuing, and agreement between parts rather than to any single part's work.

The cleanest way to separate compute from memory bandwidth is the roofline reasoning of Chapter 3. Characterize a step by its arithmetic intensity, the ratio of floating-point operations to bytes moved,

A machine has two ceilings: a peak compute rate $\pi$ (FLOP per second) and a peak memory bandwidth $\beta$ (bytes per second). The most arithmetic a step can sustain is the lower of what compute allows and what the memory feed allows,

The crossover sits at the ridge intensity $I^\star = \pi / \beta$. A step with $I < I^\star$ lives on the slanted part of the roofline, starved by bandwidth: it is memory-bandwidth bound, and on one machine the fix is to shrink the footprint, which on a cluster means distributing the model. A step with $I > I^\star$ lives under the flat part, limited by the arithmetic units: it is compute bound, and the fix is to add compute, which means distributing training. The two simplest binding tests fall straight out:

Communication adds a third roof. When a step also moves $Q_{\text{net}}$ bytes between machines over a link of bandwidth $\beta_{\text{net}}$, with the per-message latency $\alpha$ of the alpha-beta model from Chapter 3, a single round costs about $\alpha + Q_{\text{net}}/\beta_{\text{net}}$, and the step is communication bound when that time dominates the local work,

When the communication roof is the binding one, no amount of extra compute or memory helps; the remedy lives on the coordination axis, and adding workers can make matters worse by raising $Q_{\text{net}}$. Figure 41.2.1 collapses these tests into the diagnostic flow you actually run, and the demo that follows it puts numbers under each branch.

3. A Profiler That Picks the Axis for You Intermediate

The diagnosis in Figure 41.2.1 is only as good as the profile underneath it, so the right first move on any capstone is to time the candidate steps of the single-machine baseline and compute their arithmetic intensity. The demonstration below builds three deliberately different steps, a compute-heavy dense matrix multiply, a memory-heavy streaming reduction over a large array, and an I/O-heavy write-then-read against disk, then times each and reports its arithmetic intensity and sustained throughput. The classification falls out of those numbers: a high FLOP-per-byte step that nonetheless sustains few gigabytes per second is compute bound, a low FLOP-per-byte step that saturates memory bandwidth is bandwidth bound, and a step dominated by bytes crossing the disk boundary is I/O bound. Each verdict maps to a row of Table 41.2.1.

import time, numpy as np

def timed(fn, reps=3):
    best = float("inf")
    for _ in range(reps):
        t0 = time.perf_counter(); fn(); best = min(best, time.perf_counter() - t0)
    return best

rng = np.random.default_rng(0)

# Workload A: compute-bound. Dense matmul: high FLOP per byte.
A = rng.standard_normal((1200, 1200)); B = rng.standard_normal((1200, 1200))
compute_step = lambda: A @ B
flops_A, bytes_A = 2.0 * 1200**3, 3 * A.nbytes        # ~2 n^3 flops; read A, read B, write C

# Workload B: memory-bandwidth-bound. Stream a 305 MB array, do almost no arithmetic.
big = rng.standard_normal(40_000_000)
memory_step = lambda: big.sum()
flops_B, bytes_B = 2.0 * big.size, big.nbytes          # ~2 flops/elem; one streaming read

# Workload C: I/O-bound. Write then read a 61 MB file from disk.
import os, tempfile
payload = rng.standard_normal(8_000_000)
path = os.path.join(tempfile.gettempdir(), "_bn_io_probe.bin")
def io_step():
    payload.tofile(path)
    with open(path, "rb") as f: f.read()
flops_C, bytes_C = 0.0, 2 * payload.nbytes             # one write + one read

def classify(name, t, flops, byts, axis):
    I = flops / byts if byts else 0.0                  # arithmetic intensity, FLOP per byte
    print(f"{name:14s} t={t*1e3:7.1f} ms  I={I:8.2f} F/B  "
          f"{flops/t/1e9:7.1f} GFLOP/s  {byts/t/1e9:6.2f} GB/s  -> {axis}")

print("workload        time          arith.intensity   throughput          recommended axis")
print("-" * 92)
classify("compute-heavy", timed(compute_step), flops_A, bytes_A, "distribute training (split compute)")
classify("memory-heavy ", timed(memory_step), flops_B, bytes_B, "distribute the model (split footprint)")
classify("io-heavy     ", timed(io_step),     flops_C, bytes_C, "distribute data (shard the dataset)")
os.remove(path)

workload        time          arith.intensity   throughput          recommended axis
--------------------------------------------------------------------------------------------
compute-heavy  t= 2495.4 ms  I=  100.00 F/B      1.4 GFLOP/s    0.01 GB/s  -> distribute training (split compute)
memory-heavy   t=   47.6 ms  I=    0.25 F/B      1.7 GFLOP/s    6.73 GB/s  -> distribute the model (split footprint)
io-heavy       t=   72.4 ms  I=    0.00 F/B      0.0 GFLOP/s    1.77 GB/s  -> distribute data (shard the dataset)

4. Distributing the Wrong Axis Advanced

The failure mode this section exists to prevent is mechanical and common: a team feels their system is slow, reaches for the most familiar form of distribution, and applies it to a ceiling it cannot lift. The canonical example is data parallelism aimed at a memory-capacity problem. Data-parallel training, the exact-gradient method of Chapter 15, replicates the full model on every worker and splits the batch among them. If the model already does not fit on one device, replicating it puts the same too-large model on each of $K$ workers, every one of which fails to allocate exactly as the single machine did. You have multiplied the cost by $K$ and the capacity by nothing. The memory-capacity ceiling is lifted only by splitting the parameters and activations themselves, the model-parallel and sharded methods of Chapter 16.

The symmetric error runs the other way. A model that fits comfortably and is purely compute bound gains nothing from model parallelism; splitting it across devices adds the activation-passing communication that the roofline test above would have flagged, slowing a step that needed only more arithmetic units thrown at it in parallel. The same logic disqualifies inference replication for a training-compute problem, and data sharding for a system whose dataset already fits in memory and whose pain is request latency. Each mismatch shares one shape: a remedy whose ceiling was never the binding one, paying the communication tax of Chapter 3 for no return. The diagnosis in Figure 41.2.1 is precisely the guard against this, which is why it precedes any commitment to an architecture.

5. From Verdict to Project Intermediate

Once the profile has named the binding ceiling and Table 41.2.1 has named the axis, the capstone has a spine. The axis fixes which part of the book you mine for technique, which production tool you adopt, and which metric defines success, so the rest of the design becomes a matter of following one column down rather than surveying the whole field. A capstone is rarely a single axis forever; real systems eventually stack several, as a large-model project distributes data, training, the model, and cluster coordination at once. But a capstone earns the right to a second axis only by saturating the first. You distribute the binding ceiling, re-profile, and let the next-tightest ceiling, now revealed, justify the next axis. Choosing one axis at a time, each one forced by a fresh measurement, is what keeps the project honest and finishable.

With the axis named and defended, the project can move from diagnosis to construction. The next step is to build and measure the single-machine baseline properly, the artifact you profiled informally here becomes the rigorous reference point that every distributed speedup is measured against, in Section 41.3. The architecture that distributes the chosen axis, with its concrete partitioning, communication pattern, and fault-tolerance plan, is committed in Section 41.4.