Practical Bottlenecks and Scaling Efficiency

"They added thirty-two of me and expected thirty-two times the work. Instead I spent my afternoons waiting on the wire, perfectly synchronized and almost entirely idle."

A GPU Idling on a Communication Barrier

Every prior section of this chapter added one mechanism that makes data-parallel training fast: the exact gradient average (Section 15.3), the all-reduce that combines it (Section 15.4), the bucketing that overlaps communication with the backward pass (Section 15.5), the framework that wires it together (Section 15.6), and the mixed precision that shrinks both the compute and the bytes on the wire (Section 15.8). Each one works. Yet a real job almost never hits the linear speedup the gradient identity promises, and the gap between the speedup you expected and the speedup you got is where a practitioner earns their keep. This section is a diagnosis manual. We catalog the usual suspects, give you the two numbers that localize the fault, and state plainly the conditions under which no amount of tuning will save data parallelism and a different parallelism axis is the only way forward.

1. The Anatomy of a Training Step Beginner

To reason about why scaling stalls, hold a clear picture of where time goes inside one optimizer step. Under strong scaling, the regime this section assumes throughout, the global batch is fixed and we add workers to finish each step faster; the per-GPU batch therefore shrinks as the worker count $K$ grows. Three costs run in every step. The first is compute: each worker performs the forward and backward pass on its local shard of the batch, and this time falls roughly as $1/K$ because each worker sees fewer examples. The second is communication: the gradient all-reduce, whose cost per worker for a ring algorithm is close to constant in $K$ (each worker sends and receives about twice the gradient payload regardless of how many peers it has). The third is data loading: the input pipeline must deliver the local batch to each GPU before the step can begin, a cost studied in depth in Section 8.5.

The reason data parallelism scales at all is overlap. As Section 15.5 showed, a good implementation fires the all-reduce for early gradient buckets while the backward pass is still computing later ones, and prefetches the next batch while the current one trains. When communication and loading both hide completely behind compute, the step time is just the compute time and you get linear speedup. The trouble begins when compute shrinks (because $K$ grew) below the fixed cost of communication or loading: now those costs are exposed, the GPU sits idle waiting on the wire or the disk, and adding workers stops helping. Figure 15.9.1 shows this transition as the bend in the scaling-efficiency curve.

2. The Usual Suspects Beginner

When a data-parallel job misses linear scaling, the cause is almost always one of five recurring failures. Naming them is half the diagnosis, because each has a distinct signature in the profiler and a distinct remedy. Table 15.9.1 lists them with the chapter or section that develops the underlying mechanism.

The first three are systems failures: the hardware is idle and the fix is to keep it busy. The exposed all-reduce is the most common at scale, because its per-worker cost is roughly constant while per-worker compute shrinks with $K$; eventually the communication pokes out from behind the backward pass and the curve bends, exactly as in Figure 15.9.1. A starved input pipeline is the most embarrassing, because it has nothing to do with distribution at all: the GPUs are waiting on a slow disk or an under-provisioned set of loader processes, and you are paying for accelerators to watch a progress bar. Stragglers turn the synchronous all-reduce into a barrier set to the speed of the slowest worker, so a single hot GPU, a throttled NIC, or one rank that drew a batch of unusually long sequences drags the entire group down, a coordination problem first framed in Section 2.7.

The last two suspects are subtler because they are not about idle hardware. A tiny per-GPU batch keeps the accelerator nominally busy but at terrible efficiency: with too few examples per step the matrix multiplications are too small to saturate the tensor cores, arithmetic intensity drops, and the roofline analysis of Section 3.7 tells you the GPU is running memory-bound at a fraction of its peak. Worst of all is the large-batch optimization breakdown, the only suspect that is invisible to a systems profiler: strong scaling with a fixed per-GPU batch means the global batch grows with $K$, and past the critical batch size of Section 10.8, each additional example contributes diminishing gradient information. The job runs fast and the loss curve quietly gets worse. You can be perfectly efficient in wall-clock terms and still be wasting the compute.

3. Measuring Scaling Efficiency and the Comm-to-Compute Ratio Intermediate

Diagnosis needs numbers, and two are enough to localize almost any scaling problem. The first is scaling efficiency, the formal version of "are we getting our money's worth from these GPUs." For a strong-scaling run with throughput $T(K)$ on $K$ workers, the speedup is $S(K) = T(K)/T(1)$ and the efficiency is

Perfect linear scaling gives $E(K) = 1$; an efficiency of $0.6$ means six of every ten GPU-hours you are paying for turn into training and four are lost to communication, idle waiting, or underutilization. Plotting $E(K)$ against $K$ produces the curve in Figure 15.9.1, and the worker count where it falls below your tolerance (commonly $0.8$) is the practical limit of data parallelism for that job on that cluster. This metric and the broader vocabulary for it are developed as part of the evaluation toolkit in Section 5.4; here we use it as the headline diagnostic.

The second number tells you why the efficiency fell. It is the communication-to-computation ratio, the time spent moving gradients divided by the time spent computing them in one step:

While $\gamma < 1$ and the implementation overlaps well, communication hides behind compute and efficiency stays near one. As $K$ grows under strong scaling, $t_{\text{compute}}$ falls like $1/K$ while $t_{\text{comm}}$ stays roughly flat, so $\gamma$ rises; the knee in the efficiency curve is precisely where $\gamma$ crosses one and the all-reduce can no longer hide. A ratio above one is the unambiguous signature of a communication-bound job, and it points straight at the remedies: shrink the payload (mixed precision from Section 15.8, gradient compression from Section 10.8), improve overlap (Section 15.5), or stop scaling along this axis. The code below assembles the chapter's three cost models into one simulator that computes both numbers and names the dominant bottleneck as $K$ grows.

# Scaling-efficiency simulator: combine the chapter's three cost models
# (compute, overlapped all-reduce, data loading) and find the dominant
# bottleneck as the worker count K grows. Pure Python, no dependencies.

P = 1.0e9            # parameters in the gradient
bytes_per_param = 2  # bf16 gradient payload
B_global = 8192      # fixed global batch (strong scaling)
flops_per_ex = 6 * P # rough fwd+bwd FLOPs per example for a P-param model
gpu_flops = 60e12    # sustained 60 TFLOP/s per GPU
net_GBps = 25.0      # 25 GB/s effective per-link bandwidth
loader_ex_per_s = 12000.0  # input pipeline throughput PER WORKER (examples/s)

def step_ms(K):
    b = B_global / K                                   # per-GPU local batch
    t_compute = (b * flops_per_ex / gpu_flops) * 1e3   # local fwd+bwd
    payload = P * bytes_per_param
    t_comm = (2.0 * (K - 1) / K) * payload / (net_GBps * 1e9) * 1e3  # ring all-reduce
    t_load = (b / loader_ex_per_s) * 1e3               # deliver b examples
    # Overlap: comm and loading hide BEHIND compute up to the compute budget.
    exposed_comm = max(0.0, t_comm - t_compute)
    exposed_load = max(0.0, t_load - t_compute)
    t_step = t_compute + max(exposed_comm, exposed_load)
    if exposed_comm == 0.0 and exposed_load == 0.0:
        dom = "compute-bound (overlap wins)"
    elif exposed_comm >= exposed_load:
        dom = "communication-bound"
    else:
        dom = "data-loading-bound"
    return t_compute, t_comm, t_load, t_step, dom

t1 = step_ms(1)[3]   # single-worker step time, the baseline
print(f"{'K':>5} {'compute':>9} {'comm':>8} {'load':>8} {'step':>9} "
      f"{'speedup':>8} {'eff%':>6}  dominant bottleneck")
print("-" * 78)
for K in [1, 2, 4, 8, 16, 32, 64, 128, 256]:
    tc, tk, tl, ts, dom = step_ms(K)
    speedup = t1 / ts
    eff = 100.0 * speedup / K
    print(f"{K:>5} {tc:>8.1f} {tk:>8.1f} {tl:>8.1f} {ts:>8.1f} "
          f"{speedup:>7.1f}x {eff:>5.0f}  {dom}")

    K   compute     comm     load      step  speedup   eff%  dominant bottleneck
------------------------------------------------------------------------------
    1    819.2      0.0    682.7    819.2     1.0x   100  compute-bound (overlap wins)
    2    409.6     80.0    341.3    409.6     2.0x   100  compute-bound (overlap wins)
    4    204.8    120.0    170.7    204.8     4.0x   100  compute-bound (overlap wins)
    8    102.4    140.0     85.3    140.0     5.9x    73  communication-bound
   16     51.2    150.0     42.7    150.0     5.5x    34  communication-bound
   32     25.6    155.0     21.3    155.0     5.3x    17  communication-bound
   64     12.8    157.5     10.7    157.5     5.2x     8  communication-bound
  128      6.4    158.8      5.3    158.8     5.2x     4  communication-bound
  256      3.2    159.4      2.7    159.4     5.1x     2  communication-bound

The simulator makes the lesson concrete: the speedup saturates near $5.9\times$ no matter how many workers you add past the knee, because the step time can never fall below the fixed all-reduce cost. This is Amdahl's law wearing a network cap, and it is the quantitative statement of why data parallelism has a ceiling. The exact location of the knee depends on your model size, interconnect, and per-GPU batch, but the shape is universal, and every production training team learns to find their own knee before committing to a worker count.

import torch
from torch.profiler import profile, ProfilerActivity, schedule

with profile(
        activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
        schedule=schedule(wait=1, warmup=1, active=3),  # skip warmup steps
        record_shapes=True, with_stack=True) as prof:
    for step, batch in enumerate(loader):
        train_step(model, batch)   # forward, backward, optimizer.step
        prof.step()

# Sort by CUDA time: NCCL all-reduce kernels rising to the top is the
# tell-tale of a communication-bound job whose overlap has broken down.
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=12))
prof.export_chrome_trace("trace.json")   # open in chrome://tracing or Perfetto

4. The Verdict on Data Parallelism Intermediate

Data parallelism is the workhorse of distributed deep learning, and for good reason: it is exact (Section 15.3), it is simple to wrap around an existing model (Section 15.6), and through its compute-bound region it scales linearly with almost no effort. For the great majority of training jobs, a model that fits in one accelerator's memory and a worker count below the communication knee, it is not just adequate but optimal, and reaching for anything fancier would add complexity for no gain. The honest verdict is that data parallelism is excellent until one of two limits bites.

The first limit is the communication wall this section has measured: past the knee, the exposed all-reduce caps your speedup no matter how many workers you add, and the only escape is to move less data per step or to overlap it differently. The second limit is harder and is the one Chapter 16 exists to address: data parallelism replicates the entire model on every worker, so it does nothing at all for a model too large to fit on one device. When the parameters, optimizer state, and activations exceed a single accelerator's memory, you cannot run even a single data-parallel replica, and no number of workers fixes that. At that point you must partition the model itself, with the sharded, pipeline, and tensor-parallel methods of Chapter 16, often combined with data parallelism into the hybrid strategies that train today's largest models.

5. Chapter Summary Beginner

Chapter 15 took data parallelism from a single equation to a production training loop and then to its limits. We began with why deep learning needs distributed training, climbed from single-GPU to multi-node execution, and grounded the whole method in the exact gradient average that makes splitting a batch across $K$ workers a reorganization rather than an approximation. We built the all-reduce that combines the partial gradients, the bucketing and overlap that hide its cost behind the backward pass, and the frameworks (PyTorch DDP and Horovod) that wire it all together in a few lines. We added mixed precision as a per-node enabler that shrinks both the compute and the bytes on the wire. This closing section synthesized all of it into a single diagnostic discipline: measure scaling efficiency $E(K)$ and the comm-to-compute ratio $\gamma(K)$, name which of the five suspects dominates, and recognize the knee where data parallelism gives way to the next axis.