What Does It Mean to Scale?

"They added a hundred more of me and were shocked that I went only twice as fast. Nobody asked the one part of the job that refuses to be cloned."

A Worker That Read Amdahl's Fine Print

Every engineer has heard the phrase "it scales" and every engineer has watched a system that supposedly scaled grind to a near-standstill once a few dozen machines were involved. The word carries too much hope and too little precision. Before this chapter can model where speedups come from and where they leak away, we need a definition of scaling sharp enough to put on an axis and plot. That definition starts from a single measurable thing: how long the job takes when you run it on $K$ machines instead of one. From that one quantity we derive everything else, and we ground each derivation in the workload this book cares about most, a training step distributed across $K$ GPUs.

1. Speedup: How Much Faster Did We Go? Beginner

Let $T(K)$ be the wall-clock time to finish a fixed job using $K$ workers, so $T(1)$ is the single-machine baseline. The speedup on $K$ workers is the ratio of the baseline time to the parallel time,

Speedup is a pure number that answers the only question a practitioner truly cares about: did adding machines make the job finish sooner, and by what factor? If a training epoch takes 80 minutes on one GPU and 20 minutes on four, then $S(4) = 80/20 = 4$. Concretely, picture the job as processing a fixed number of training steps, say one epoch over a fixed dataset. On one GPU the epoch streams through every minibatch in sequence; on $K$ GPUs running data-parallel training, each GPU owns a shard of the minibatches, so the per-step throughput (minibatches finished per second across the whole cluster) is what $S(K)$ ultimately measures. The baseline $T(1)$ must be the best honest single-machine time, not a deliberately crippled one, or every later number is inflated.

The aspirational case is linear or ideal speedup, $S(K) = K$: four GPUs finish in a quarter of the time, a thousand GPUs in a thousandth. Real systems almost always fall short, giving sublinear speedup, $S(K) < K$, because some part of the work does not parallelize and because the workers must spend time communicating. The gap between the line $S(K) = K$ and the curve a real system traces is the entire subject of this chapter, and the all-reduce of Chapter 4 is the operation that usually opens it.

2. Efficiency: How Much of Each Machine Did We Waste? Beginner

Speedup alone flatters large clusters: a speedup of 50 sounds triumphant until you learn it took 200 GPUs to get there. Efficiency normalizes speedup by the resources spent,

Efficiency lives in $[0, 1]$ (occasionally above, as Section 3 notes) and answers a budgeting question: of the $K$ machines you are paying for, what fraction is doing useful work rather than waiting? Ideal speedup $S(K) = K$ corresponds to $E(K) = 1$, every machine fully earning its keep. A speedup of 50 on 200 GPUs is $E = 0.25$: three quarters of the fleet is, in effect, idling on barriers and shuffling gradients. For a data-parallel training step, low efficiency usually means the per-step all-reduce has grown to rival the per-step compute, so the GPUs spend their time waiting on the network rather than on matrix multiplies. Efficiency becomes a reporting standard for system papers in Chapter 5; here it is simply the honest companion that keeps speedup from lying.

3. Scalability: How Does Efficiency Behave as K Grows? Intermediate

A single $(S, E)$ pair at one value of $K$ tells you about one cluster size. Scalability is the trend: how $E(K)$ behaves as $K$ increases. A system is said to scale well if efficiency stays high as you add machines, and to scale poorly if efficiency collapses. Because some part of almost every job is inherently serial and communication grows with $K$, the typical curve is monotonically decreasing: $E(1) = 1$, then a gentle decline, then a cliff. Plotting $S(K)$ against $K$ on the same axes as the ideal line $S(K) = K$ makes the erosion visible at a glance, which is exactly what Figure 3.1.1 does.

The flattening in Figure 3.1.1 is not an accident of one bad system; it is the generic fate of a fixed job spread ever more thinly. Some fraction of the work, loading the first batch, the optimizer step that touches the whole model, a logging barrier, simply will not run faster no matter how many GPUs wait on it, and that fraction sets the horizontal ceiling the real curve approaches. The next section makes the curve numeric so you can see the ceiling arrive.

4. A Runnable Speedup and Efficiency Table Intermediate

To see scalability turn into numbers, model a training job with a small serial fraction $f$: the portion of the per-step work that cannot be parallelized across GPUs. If the parallel portion $1 - f$ shrinks by a factor of $K$ while the serial portion $f$ stays fixed, then $T(K) = f + (1 - f)/K$ with $T(1) = 1$, and the speedup follows directly. The code below sweeps $K$ from $1$ to $512$ for $f = 0.05$ (a realistic 5% serial step on a well-tuned data-parallel job) and prints $T(K)$, $S(K)$, and $E(K)$ in one pass.

serial_fraction = 0.05  # 5% of the work cannot be parallelized across GPUs

def speedup(K, f):
    # T(1)=1; parallel part 1-f shrinks by K, serial part f stays fixed
    return 1.0 / (f + (1.0 - f) / K)

print(f"serial fraction f = {serial_fraction}")
print(f"{'K':>6} {'T(K)':>10} {'S(K)':>10} {'E(K)':>10}")
for K in [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]:
    S = speedup(K, serial_fraction)
    T = 1.0 / S
    E = S / K
    print(f"{K:>6} {T:>10.4f} {S:>10.4f} {E:>10.4f}")

serial fraction f = 0.05
     K       T(K)       S(K)       E(K)
     1     1.0000     1.0000     1.0000
     2     0.5250     1.9048     0.9524
     4     0.2875     3.4783     0.8696
     8     0.1688     5.9259     0.7407
    16     0.1094     9.1429     0.5714
    32     0.0797    12.5490     0.3922
    64     0.0648    15.4217     0.2410
   128     0.0574    17.4150     0.1361
   256     0.0537    18.6182     0.0727
   512     0.0519    19.2844     0.0377

The table is Figure 3.1.1 made arithmetic. For $K \le 4$ the speedup is nearly linear and efficiency stays above $0.85$; this is the regime where adding GPUs is almost free. By $K = 32$ efficiency has fallen below $0.4$, meaning more than half the fleet is idle, and by $K = 512$ each added GPU returns almost nothing while still drawing power and rent. The horizontal ceiling that Figure 3.1.1 drew is the limit $S(\infty) = 1/f = 20$, reached here from below. This is the central lesson of scalability stated in numbers: a small unparallelizable fraction sets a hard ceiling on speedup, and efficiency is the early-warning gauge that tells you the ceiling is near long before the speedup curve visibly flattens. Where that $1/f$ ceiling comes from formally is the business of Section 3.5.

5. Superlinear Speedup: When the Curve Beats the Line Advanced

Occasionally a measured system reports $S(K) > K$, efficiency above $1$, a curve that rises above the ideal diagonal of Figure 3.1.1. This superlinear speedup looks like a free lunch and is almost never one. The usual cause is the memory hierarchy: a dataset or working set that overflows one machine's cache (or spills from GPU memory to host memory) fits comfortably once it is sharded across $K$ machines, so each worker suddenly runs from a faster level of the hierarchy. The extra speed is real, but it comes from a change in where the data lives, not from the parallelism itself, so it is a cache or memory-pressure artifact rather than a law you can extrapolate. Treat a superlinear report as a prompt to ask which resource the single-machine baseline was starved of; an honest $T(1)$ on adequate memory usually removes the effect.

6. Scaling a Fixed Problem vs Scaling the Problem With Resources Intermediate

Everything above measured speedup on a fixed job: the dataset, the model, and the work were held constant while $K$ grew. This is strong scaling, and the table in Output 3.1.1 is a strong-scaling study. Strong scaling answers "can I finish the same job faster by adding machines?" and, as the serial fraction shows, it has a hard ceiling. But it is not the only thing people mean by "scaling", and conflating the two is a common source of argument.

The other regime grows the problem along with the resources. If you double the GPUs and also double the global batch size or the dataset, you are asking a different question: "can I do a proportionally bigger job in the same time?" This is weak scaling, and because the per-worker work stays constant, it often holds efficiency far higher than strong scaling does. Training foundation models is overwhelmingly a weak-scaling story: nobody buys a thousand GPUs to train the same tiny model faster; they buy them to train a vastly larger model on vastly more data within a fixed wall-clock budget. The distinction is important enough that Section 3.3 is devoted to it, and the optimism of Gustafson's law in Section 3.5 rests entirely on the weak-scaling view. For now, fix the vocabulary: strong scaling holds the problem fixed and chases speed, weak scaling grows the problem and chases size, and the same cluster can look like a failure under one lens and a triumph under the other.

import torch.utils.benchmark as bench

# Time the same training step on 1 GPU and on K GPUs (run each config separately).
t = bench.Timer(stmt="train_one_step(model, batch)",
                globals={"train_one_step": train_one_step, "model": model, "batch": batch})
median_seconds = t.blocked_autorange().median   # robust per-step time, warmup handled

# Then S(K) = t1.median / tK.median and E(K) = S(K) / K, computed from measured times.

7. From Definitions to Models Beginner

We now have the three quantities the rest of the chapter manipulates. Speedup $S(K) = T(1)/T(K)$ says how much faster a fixed job ran; efficiency $E(K) = S(K)/K$ says how much of each machine was wasted; and scalability is the shape of $E(K)$ as $K$ climbs, a shape that bends and flattens whenever a serial fraction or communication cost refuses to shrink. We distinguished strong scaling (fixed problem, chasing speed, hard ceiling) from weak scaling (problem grows with resources, chasing size, far gentler erosion), and we learned to read efficiency above $1$ as a baseline smell rather than a victory. Every later model in this chapter is a way to predict the curve in Figure 3.1.1 from the structure of a workload rather than measuring it after the fact. The most direct of those models, the one that turns the serial fraction $f$ into the ceiling $1/f$ we watched the table approach, is Amdahl's law, and its optimistic weak-scaling counterpart is Gustafson's law; both wait in Section 3.5. The next section stays closer to the hardware, separating the two ways to add capacity, scaling out across more machines and scaling up within one, in Section 3.2.