Communication-to-Computation Ratio

"They kept adding workers and asking me to go faster. I kept spending more of my day on the phone. Nobody asked which of those two facts caused the other."

A GPU Idling on a Communication Barrier

The previous sections of this chapter gave you outcome metrics. Speedup and efficiency say how far a job is from linear scaling; goodput and tail latency say how much useful work survives contention and stragglers. Those are the symptoms. To treat the disease you need a metric that points at the mechanism, and for distributed training the dominant mechanism is the cost of moving gradients between machines. Chapter 4 opens with the thesis that communication, not computation, is what ultimately bounds large-scale training; this section turns that thesis into a quantity you can read off a profiler and act on. The communication-to-computation ratio, written $R$ throughout, is the single most useful diagnostic in your evaluation toolkit precisely because it converts a vague complaint ("it does not scale") into a measured cause.

1. Defining the Ratio Beginner

Decompose one step of a distributed worker into the time it spends on useful arithmetic and the time it spends moving data. Let $T_{\text{comp}}$ be the compute time of a step (forward and backward passes, the actual floating-point work) and let $T_{\text{comm}}$ be the communication time exposed on the critical path, meaning the network time that is not hidden behind compute. The communication-to-computation ratio is their quotient,

The wall-clock time of a step, when communication is not overlapped, is $T_{\text{step}} = T_{\text{comp}} + T_{\text{comm}}$. The fraction of that step doing useful work is the per-step scaling efficiency,

This little identity is the whole reason the ratio matters. Efficiency is a falling function of $R$: at $R = 0$ the step is pure compute and $E = 100\%$; at $R = 1$ the worker splits its time evenly and $E = 50\%$; at $R = 4$ it spends four fifths of every step on the network and $E = 20\%$. The word "exposed" in the definition of $T_{\text{comm}}$ is load-bearing, and Section 5 returns to it: modern frameworks overlap much of the all-reduce with the backward pass, so the number that belongs in $R$ is the communication that overlap failed to hide, not the raw transfer time.

2. Why the Ratio Grows With Scale Intermediate

The ratio is not a constant of the job; it grows as you add workers, which is exactly why scaling stalls. The communication-cost model of Section 3.8 gives the shape. For a ring all-reduce of a gradient with $P$ parameters across $K$ workers, the bytes each worker sends are roughly $2(K-1)/K$ times the gradient size, and under the $\alpha$-$\beta$ model the time is approximately

where $\alpha$ is the per-message latency and $\beta$ is the per-byte transfer cost. The bandwidth term saturates as $K$ grows (the factor $(K-1)/K$ approaches one), but the latency term $2(K-1)\alpha$ keeps climbing linearly in $K$. Meanwhile, if you hold the global batch fixed and add workers, each worker's compute $T_{\text{comp}}$ shrinks because it processes fewer examples. A growing numerator over a shrinking denominator is a ratio that rises fast, which is the analytic reason that fixed-batch strong scaling hits a wall. Chapter 4 develops this bound for every collective; here the point is only that $R$ is a function of scale, so you must measure it at the worker count you actually run.

3. Measuring It From a Real Step Intermediate

You do not estimate $R$ from a formula in practice; you measure it. The procedure is always the same: profile one steady-state training step, attribute each slice of wall-clock to compute or to communication, then take the quotient. Three tools cover almost every case. The PyTorch profiler (torch.profiler) tags each operator as compute or as an NCCL collective and gives you a per-category time breakdown directly. CUDA events placed around the backward pass and around the all-reduce give you the two raw numbers with microsecond resolution when you want a lightweight in-loop measurement. And nccl-tests, run standalone on the same interconnect, gives you the pure collective time for a gradient of your size, which is the $T_{\text{comm}}$ you would see with zero overlap. Read all three the same way: the only quantities you need are the compute time and the communication time left on the critical path.

import torch
from torch.profiler import profile, ProfilerActivity

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
    train_one_step(model, batch)          # forward, backward, all-reduce, optimizer
    torch.cuda.synchronize()

# Group kernels: NCCL collectives are communication, everything else is compute.
ev = prof.key_averages()
comm = sum(e.cuda_time_total for e in ev if "nccl" in e.key.lower())
comp = sum(e.cuda_time_total for e in ev) - comm
print(f"R = {comm / comp:.3f}   efficiency = {comp / (comp + comm) * 100:.1f}%")

The runnable demo below skips the GPU and works directly from a profiled breakdown, because the arithmetic of the ratio is what we want to make concrete, not the act of capturing a trace. Given a compute time and a raw all-reduce time, it computes $R$ and the resulting efficiency, then shows how the two levers of Section 4.10 (overlap) and gradient compression move both numbers. These are the same numbers plotted in Figure 5.4.1.

def scaling_efficiency(t_compute, t_exposed_comm):
    """Fraction of step time spent on useful compute; the rest is comm stall."""
    step = t_compute + t_exposed_comm
    ratio = t_exposed_comm / t_compute          # communication-to-computation ratio
    eff = t_compute / step                      # = 1 / (1 + ratio)
    return ratio, eff


# Measured per-step times in milliseconds, from a profiler breakdown of one
# training step on a worker: compute (forward + backward) and the raw all-reduce.
t_compute = 180.0          # ms of useful FLOPs per step
t_comm_raw = 120.0         # ms the all-reduce takes if nothing hides it

print("Baseline: no overlap, no compression")
ratio, eff = scaling_efficiency(t_compute, t_comm_raw)
print(f"  exposed comm        : {t_comm_raw:6.1f} ms")
print(f"  comm/compute ratio  : {ratio:6.3f}")
print(f"  scaling efficiency  : {eff*100:5.1f} %")

print("\nWith overlap: 70% of the all-reduce hides behind the backward pass")
overlap = 0.70
t_comm_overlapped = t_comm_raw * (1.0 - overlap)
ratio, eff = scaling_efficiency(t_compute, t_comm_overlapped)
print(f"  exposed comm        : {t_comm_overlapped:6.1f} ms")
print(f"  comm/compute ratio  : {ratio:6.3f}")
print(f"  scaling efficiency  : {eff*100:5.1f} %")

print("\nWith overlap + 4x gradient compression on the exposed remainder")
t_comm_compressed = t_comm_overlapped / 4.0
ratio, eff = scaling_efficiency(t_compute, t_comm_compressed)
print(f"  exposed comm        : {t_comm_compressed:6.1f} ms")
print(f"  comm/compute ratio  : {ratio:6.3f}")
print(f"  scaling efficiency  : {eff*100:5.1f} %")

Baseline: no overlap, no compression
  exposed comm        :  120.0 ms
  comm/compute ratio  :  0.667
  scaling efficiency  :  60.0 %

With overlap: 70% of the all-reduce hides behind the backward pass
  exposed comm        :   36.0 ms
  comm/compute ratio  :  0.200
  scaling efficiency  :  83.3 %

With overlap + 4x gradient compression on the exposed remainder
  exposed comm        :    9.0 ms
  comm/compute ratio  :  0.050
  scaling efficiency  :  95.2 %

4. Reading the Ratio: A Decision Rule Intermediate

The value of $R$ tells you which lever to pull, and pulling the wrong one wastes money. Table 5.4.1 is the rule the rest of this book quietly follows whenever it sizes a job. The guiding principle is simple: when $R$ is small you are compute-bound and the cluster will reward more hardware, so scale out; when $R$ is large you are communication-bound and more hardware only adds more waiting, so first attack the communication term itself.

Map the four levers onto the formula and each one becomes obvious. Adding workers lowers $T_{\text{comp}}$ per worker (fixed global batch) and raises $T_{\text{comm}}$, so it increases $R$ and is only safe when $R$ starts small. A bigger local batch raises $T_{\text{comp}}$ while leaving the gradient size, and hence $T_{\text{comm}}$, unchanged, so it decreases $R$ directly; this is why large-batch training is the first reflex of anyone fighting a communication wall. Gradient compression cuts the bytes in $T_{\text{comm}}$ and so cuts $R$ at the source. Overlap cuts the exposed fraction of $T_{\text{comm}}$ without touching the bytes at all, which is why it is the cheapest lever and the one to exhaust first.

5. Overlap, and Why "Exposed" Is the Word That Matters Advanced

Everything above hinges on $T_{\text{comm}}$ being the communication left on the critical path, not the raw transfer time. The reason is the overlap trick of Section 4.10: gradients become ready layer by layer during the backward pass, so a framework can start the all-reduce of the last layer's gradients while still computing the earlier layers' gradients. The network work then happens under the compute, and only the part that does not fit behind compute is exposed. If the raw all-reduce takes $120$ ms and overlap hides $70\%$ of it, the exposed term is $36$ ms, and that $36$ is what belongs in $R$, as Output 5.4.1 shows. Using the raw $120$ ms instead would tell you the job is communication-bound when it is comfortably compute-bound, and you would reach for the wrong lever.

This is also why the ratio and the overlap trick must be read together. Overlap does not reduce the bytes moved, so it does not change the bandwidth term of the cost model; it changes only how much of that term lands on the critical path. There is a ceiling: once compute is fully packed with hidden communication, any further all-reduce time spills into the open and overlap can hide no more. Past that point the only remaining moves are the ones that touch the bytes themselves, compression and a larger local batch, or the ones that change how often you synchronize at all, which is where local-update methods enter. Measuring $R$ with the exposed term tells you exactly where on this progression a job sits.

The ratio is the diagnostic; the next section turns the time you measure into money and energy. A worker that spends 40% of every step waiting on the network is not just slow, it is burning power and rented GPU-hours while idle, and Section 5.5 shows how to put a dollar and a joule figure on exactly that waste, closing the loop from "it does not scale" to "here is what the stall costs."