Why Communication, Not Compute, Bounds Distributed Training

"I finish my gradient in four milliseconds and then sit on the barrier for forty-six, waiting for everyone else to agree on the answer. I am not slow. I am punctual to a meeting that will not start."

A GPU Idling on a Communication Barrier

In Section 1.1 we proved that data-parallel training is exact: split the examples across $K$ workers, sum their partial gradients, divide by the total count, and you recover the identical single-machine gradient. That proof settled correctness and deliberately left cost for later. Later is now. The summing step, harmless when the data sat in one process, becomes a network operation when the workers are separate machines, and the time it takes does not shrink no matter how fast the workers compute. This section measures that time against the compute it gates and shows that for the models and clusters people actually train, the network usually wins.

The structure of every data-parallel step is the same two-phase rhythm, and seeing it drawn once fixes the intuition for the whole chapter. Figure 4.1.1 lays out a single step on a timeline: a short compute phase where each worker runs the forward and backward pass on its shard, followed by a longer communication phase where the workers exchange and combine their gradients before any of them can take the optimizer step.

1. The Two Halves of a Training Step Beginner

Write the wall-clock time of one synchronous data-parallel step as the sum of the two phases in Figure 4.1.1. The compute phase takes time $T_{\text{comp}}$: each worker runs the forward and backward pass over its local batch. A widely used estimate puts the cost of one training step at roughly six floating-point operations per parameter per example (two for the forward pass, four for the backward), so for a model of $P$ parameters, a local batch of $b$ examples, and an accelerator that sustains $R$ floating-point operations per second,

The communication phase takes time $T_{\text{comm}}$: the workers all-reduce the gradient, a vector of $P$ numbers. Chapter 3 modeled the cost of moving data across a network as latency plus size over bandwidth; here we keep only the bandwidth term, because for the large gradients of real models it dominates. A ring all-reduce, the algorithm Section 4.4 dissects, moves about $2(K-1)/K$ times the gradient's bytes across each worker's link. With $g$ bytes per parameter and an effective per-worker bandwidth $B$,

The contrast between these two formulas is the whole story. The compute time carries the local batch $b$ in its numerator; the communication time does not. Add more workers to a fixed global batch and each worker's $b$ falls, so $T_{\text{comp}}$ shrinks while $T_{\text{comm}}$ holds nearly constant (the factor $2(K-1)/K$ creeps up toward $2$ and then stops). Grow the model and both times scale with $P$, so their ratio stays fixed at whatever the hardware dictates. We build the cost model carefully on the foundation laid in Section 3.8; the point here is the shape, not the third decimal place.

2. The Communication-to-Computation Ratio Intermediate

Define the communication-to-computation ratio $\gamma$ as the time spent moving the gradient divided by the time spent producing it:

The algebra delivers a small surprise: the parameter count $P$ cancels. At fixed batch and worker count, doubling the model size doubles both phases and leaves the ratio untouched, so a tiny model and a giant one sit at the same point on the curve. What does move $\gamma$ is the per-worker batch $b$ (in the denominator) and the worker count $K$ (through the factor that climbs toward $2$). Shrink $b$ by spreading a fixed global batch over more workers and $\gamma$ rises in inverse proportion. The break-even point is $\gamma = 1$: below it the step is compute-bound and adding workers still helps; above it the step is communication-bound and each new worker mostly adds waiting. The honest scaling question from Section 3.5, where Amdahl's law caps speedup at the reciprocal of the serial fraction, is exactly this: the all-reduce is the serial fraction of a training step, and $\gamma$ measures how large it has grown.

The script below turns these formulas into numbers. It fixes the hardware (a fast accelerator, an NVLink-class interconnect) and a global batch, then sweeps the worker count $K$, splitting that global batch across the workers so each one's local batch shrinks as the cluster grows. It prints the compute time, the all-reduce time, and the ratio $\gamma$, flagging the rows where the step has gone communication-bound.

def compute_seconds(P, local_batch, flop_per_param_per_example, flop_rate):
    # One step costs about 6 FLOPs per parameter per example (forward + backward).
    return flop_per_param_per_example * P * local_batch / flop_rate

def allreduce_seconds(P, bytes_per_param, bandwidth, K):
    # Ring all-reduce moves about 2(K-1)/K times the gradient bytes per worker.
    return 2.0 * (K - 1) / K * (P * bytes_per_param) / bandwidth

flop_rate = 312e12   # 312 TFLOP/s sustained (modern accelerator, bf16)
bandwidth = 600e9    # 600 GB/s effective interconnect per worker (NVLink-class)
fppe = 6.0           # FLOPs per parameter per example (fwd + bwd)
bpp = 2.0            # bytes per parameter (bf16 gradient)

# Strong scaling: a FIXED global batch of 2048 examples is split across K workers,
# so each worker's local batch shrinks as we add workers. Compute per step falls,
# communication per step does not, so the ratio climbs and eventually crosses 1.
global_batch = 2048
P = 7e9              # a 7-billion-parameter model

print("Strong scaling: P = 7e9 params, global batch = 2048 split across K workers")
print(f"{'K':>6} {'local batch':>12} {'compute (ms)':>13} {'all-reduce (ms)':>16} {'gamma':>9}")
for K in [1, 4, 16, 64, 256, 1024]:
    local = max(global_batch // K, 1)
    tc = compute_seconds(P, local, fppe, flop_rate)
    ta = allreduce_seconds(P, bpp, bandwidth, K)
    flag = "  <- comm-bound" if ta > tc else ""
    print(f"{K:>6d} {local:>12d} {tc*1e3:>13.2f} {ta*1e3:>16.2f} {ta/tc:>9.3f}{flag}")

Strong scaling: P = 7e9 params, global batch = 2048 split across K workers
     K  local batch  compute (ms)  all-reduce (ms)     gamma
     1         2048        275.69             0.00     0.000
     4          512         68.92            35.00     0.508
    16          128         17.23            43.75     2.539  <- comm-bound
    64           32          4.31            45.94    10.664  <- comm-bound
   256            8          1.08            46.48    43.164  <- comm-bound
  1024            2          0.27            46.62   173.164  <- comm-bound

Read the columns side by side. The compute time falls by a factor of a thousand from one worker to a thousand workers, exactly as you would hope from spreading the batch. The all-reduce time barely changes, climbing only from $35$ to $47$ milliseconds because the factor $2(K-1)/K$ is the only thing that moves. By $K = 1024$ the workers spend over $99\%$ of each step waiting on the network and well under $1\%$ computing. That is not a pathology of these particular numbers; it is the generic fate of strong scaling, and it is why the rest of this chapter is a catalogue of techniques for making the orange block in Figure 4.1.1 shorter.

3. Why Growing the Model Does Not Save You Intermediate

A natural hope is that very large models, the ones everyone wants to train, will be compute-bound simply because they do so much arithmetic. The cancellation of $P$ in the ratio already warns against this, and the second sweep makes it concrete. Holding the worker count and local batch fixed while the parameter count grows by three orders of magnitude leaves $\gamma$ exactly constant: both the compute and the all-reduce columns scale in lockstep with $P$, so a hundred-billion-parameter model is no more compute-bound than a hundred-million-parameter one running on the same cluster shape.

The lesson of Table 4.1.1 is that model size is the wrong knob for escaping the communication wall. What sets $\gamma$ is the cluster shape (how many workers, how fast their links) and the per-worker batch, never the parameter count alone. This is also why the remedies that genuinely help are the ones that change those quantities: a fatter interconnect raises $B$, a larger per-worker batch raises $b$, gradient compression lowers the effective $g$, and overlap hides $T_{\text{comm}}$ behind $T_{\text{comp}}$ so that the two phases run at once instead of in sequence. We develop the systematic version of these levers in Chapter 10 on communication-efficient optimization, and put them to work inside a real training loop in Chapter 15 on data-parallel deep learning.

4. The Collectives This Chapter Covers Beginner

If communication is the binding constraint, then the operations that perform it deserve first-class study, and that is what the rest of this chapter provides. Every one of them is introduced through the AI computation that needs it, never as abstract networking. Table 4.1.2 is the roadmap: each collective, the training pattern that gives rise to it, and the section that develops it. You have already met the first one by hand in Section 1.1; the others are its relatives, and recognizing which collective a given parallelism strategy leans on is the single most useful lens for the chapters ahead.

Before any of those, Section 4.2 draws the substrate they run on: the distinction between point-to-point messages and collective operations, and a deliberately thin view of the interconnects (NVLink, PCIe, InfiniBand, RDMA) that supply the bandwidth $B$ in our ratio. We keep that hardware tour brief on purpose, because the goal is not to become network engineers but to reason about cost well enough to predict, before launching a job, whether it will land on the green side of $\gamma = 1$ or the orange one.

import torch
from torch.profiler import profile, ProfilerActivity

with profile(activities=[ProfilerActivity.CUDA], record_shapes=False) as prof:
    train_one_step(model, batch)            # forward, backward, and the DDP all-reduce
    torch.cuda.synchronize()                # wait for the GPU stream to drain

# Rows whose name starts with "nccl:" are the all-reduce time; the rest is compute.
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=8))

5. What This Buys Us for the Rest of the Book Intermediate

We now have a single number, $\gamma$, that decides whether a distributed training step is paced by the accelerator or by the network, and a clear picture of what moves it: the per-worker batch and the cluster shape, not the model size. That number reframes every later design decision. A choice of interconnect is a choice of $B$; a parallelism strategy is a choice of which collective runs and how large its payload is; a compression scheme lowers $g$; an overlap technique removes $T_{\text{comm}}$ from the critical path entirely. None of these make sense without first accepting that communication is the thing worth optimizing, which is what this section established.

It is worth saying plainly that the goal is not zero communication; for synchronous data parallelism the all-reduce is mandatory, because the exactness proof of Section 1.1 depends on it. The goal is to keep $\gamma$ small enough that the workers spend their time computing rather than waiting, and to know, before committing a cluster, on which side of the break-even line a planned job will fall. With the thesis fixed and the ratio in hand, the next step is to look at the wires themselves: the point-to-point and collective communication model, and the thin slice of interconnect physics we need to populate $B$. That is the subject of Section 4.2, The Communication Substrate.