Scaling Efficiency and Cost-Awareness

"They kept adding nodes to make me faster. The invoice got faster too."

A Cluster That Read Its Own Bill

In Section 3.8 we priced a single communication step with the alpha-beta model and saw that the cost of moving gradients grows with both the message size and the number of participants. That term was the last piece we needed. We can now write down the full economics of a scale-out job: take the speedup curve shaped by the serial fraction from Section 3.5, subtract the communication and straggler overhead that grows with node count, turn the result into an efficiency, and finally weigh it against the price of the hardware. The output is a dollar figure per unit of finished work, and it is the number a platform team is judged on. This section ties the chapter's performance models to that figure and shows, with a runnable model, that spending more on nodes can buy a slower-finishing, more expensive job.

1. Efficiency, Restated as a Budget Question Beginner

Recall the scaling efficiency from Section 3.3. With $T_1$ the time to finish a fixed job on one node and $T_K$ the time on $K$ nodes, the speedup is $S(K) = T_1 / T_K$ and the efficiency is

Efficiency is the fraction of each node's capacity that the job actually converts into progress. At $E(K) = 1$ every node pulls its full weight; at $E(K) = 0.5$ you are paying for two nodes but getting the work of one. Amdahl's law (Section 3.5) already told us $E(K)$ must fall as $K$ grows, because the serial fraction does not shrink and the per-node share of parallel work does. The communication model of Section 3.8 makes the fall steeper: each added node adds bytes to move and one more participant to synchronize, and the stragglers of Chapter 2 mean the synchronized step waits for the slowest of $K$ machines, an effect that worsens as $K$ rises. Put those together and a serviceable efficiency model for a synchronous data-parallel job is

where $s$ is the serial fraction and $c$ folds the per-extra-node communication and straggler cost into a single coefficient. The quadratic $K(K-1)$ term is the communication tax; it is what eventually drags efficiency toward zero no matter how cheap each node is.

The budget question now writes itself. If a node costs $p$ dollars per hour and the single-node job takes $T_1$ hours, the wall-clock on $K$ nodes is $T_K = T_1 / S(K)$, and the dollar cost of the $K$ paid nodes is

This is the cleanest statement in the section: the dollar cost of the compute is the single-node price divided by efficiency. Halving efficiency doubles the bill for the same finished work. That is why efficiency, an abstract ratio in earlier sections, is the lever a cost-aware team watches.

2. Why the Cost Curve Has a Bottom Intermediate

If compute cost were the whole story, the cheapest choice would always be $K = 1$, since efficiency is highest there and $C_{\text{nodes}}$ is smallest. Two things break that conclusion in any real deployment. First, finishing sooner has value: a nightly training job that misses its deadline, an inference fleet that cannot clear its queue, or a researcher idled waiting on a result all carry a cost of slowness that pushes $K$ upward. Second, and more concretely, a running job carries fixed overhead billed by wall-clock time, not by node count: a reserved control plane, a data-streaming pipeline feeding the workers, a held dataset cache, a license metered by the hour. Call that rate $f$ dollars per wall-clock hour. The total bill becomes

Now the two forces are explicit and they pull in opposite directions. The $K\,p$ term grows with node count and is amplified by falling efficiency; it wants $K$ small. The $f\,T_K$ term shrinks as more nodes finish the job faster; it wants $K$ large. Their sum is a U-shaped curve: cost falls while the fixed overhead is the dominant term and the parallel work still scales well, reaches a minimum, then climbs once the falling efficiency makes each added node cost more than the wall-clock it saves. The minimum of that curve is the economically optimal node count $K^{\star}$, and it is strictly below the $K$ that minimizes wall-clock, because the fastest configuration keeps adding nodes long after they have stopped paying for themselves.

3. Finding the Cost-Optimal Node Count Intermediate

The model is small enough to evaluate directly. Code 3.9.1 implements the efficiency curve, the wall-clock, and the two-term cost from Section 2, then sweeps $K$ to find both the cheapest configuration and the fastest one. It uses a serial fraction of three percent, a modest per-node communication coefficient, an on-demand node price of $\$2.10$ per hour, and a fixed overhead of $\$9.00$ per wall-clock hour for the surrounding pipeline.

import numpy as np

# Efficiency falls because a serial fraction (Amdahl, Section 3.5) and a per-node
# communication + straggler cost (Section 3.8, Chapter 2) both erode the speedup.
s = 0.03      # serial / non-parallelizable fraction
c = 0.0015    # communication + straggler cost added per extra node

price = 2.10  # dollars per node-hour (on-demand accelerator instance)
fixed = 9.00  # dollars per WALL-CLOCK hour of fixed overhead billed while the job runs
W = 1.0       # one unit of work = the job that takes 1 hour on a single node

def speedup(K):
    return 1.0 / (s + (1.0 - s) / K + c * (K - 1))

def efficiency(K):
    return speedup(K) / K

def runtime_hours(K):
    return W / speedup(K)                    # fixed job, fewer hours as K grows

def total_cost(K):                           # the two terms pull in opposite directions
    return (K * price + fixed) * runtime_hours(K)

Ks = np.arange(1, 65)
costs = np.array([total_cost(int(K)) for K in Ks])
times = np.array([runtime_hours(int(K)) for K in Ks])
k_opt, k_fastest = int(Ks[np.argmin(costs)]), int(Ks[np.argmin(times)])

print(f"{'K':>4} {'speedup':>9} {'E(K)':>7} {'time_h':>8} {'cost_$':>9}")
for K in [1, 2, 4, 8, 16, 24, 32, 48, 64]:
    print(f"{K:>4} {speedup(K):>9.2f} {efficiency(K):>7.2%} "
          f"{runtime_hours(K):>8.3f} {total_cost(K):>9.3f}")
print()
print(f"cost-optimal K        : {k_opt}  (cost = ${total_cost(k_opt):.3f}, "
      f"E = {efficiency(k_opt):.1%})")
print(f"technically-fastest K : {k_fastest}  (time = {runtime_hours(k_fastest):.3f} h, "
      f"cost = ${total_cost(k_fastest):.3f})")
print(f"price of pure speed   : ${total_cost(k_fastest) - total_cost(k_opt):.3f} "
      f"extra to go from K={k_opt} to K={k_fastest}")

   K   speedup    E(K)   time_h    cost_$
   1      1.00 100.00%    1.000    11.100
   2      1.94  96.81%    0.516     6.818
   4      3.61  90.25%    0.277     4.820
   8      6.18  77.28%    0.162     4.173
  16      8.84  55.25%    0.113     4.819
  24      9.53  39.71%    0.105     6.232
  32      9.36  29.26%    0.107     8.139
  48      8.28  17.26%    0.121    13.254
  64      7.16  11.19%    0.140    20.027

cost-optimal K        : 8  (cost = $4.173, E = 77.3%)
technically-fastest K : 25  (time = 0.105 h, cost = $6.445)
price of pure speed   : $2.272 extra to go from K=8 to K=25

The table tells the chapter's closing story number by number. Speedup keeps rising until $K = 24$ and then falls, the textbook Amdahl-plus-communication shape. Efficiency, however, is already below $80\%$ by $K = 8$ and below $30\%$ by $K = 32$: most of the fleet is idle on the barrier. Wall-clock bottoms out near $K = 25$. The cost column is the one a team is accountable for, and it bottoms out far earlier, at $K = 8$, then climbs steeply. The economically optimal node count is well below the technically maximal one, exactly as Figure 3.9.1 draws it, and the gap between them is money spent to buy a wall-clock improvement that the fixed overhead did not justify.

4. From Node Cost to Cost-per-Token Advanced

The same arithmetic governs the unit that dominates modern AI budgets: cost per token. For a serving fleet the work is not a single job but a stream of requests, and the natural denominator is tokens generated. If the fleet costs $C$ dollars per hour and produces $G$ tokens per hour, the unit cost is $C / G$, and $G$ is itself $K$ times the per-node token rate scaled by efficiency. The structure is identical to Section 1: throughput per dollar is the per-node rate multiplied by $E(K)$, so the same falling-efficiency curve that inflated the cost of a training job inflates the cost of every token a degraded fleet emits. This is why cost-aware AI treats efficiency and utilization as first-class production metrics rather than performance trivia: at scale, a ten-point drop in efficiency is a ten-percent rise in the price of the product. The serving side of this story, where batching and the KV cache set the per-node token rate before the fleet multiplies it, is the subject of Chapter 22 and Chapter 24.

# Wrap a few training steps; the trace separates compute from comm/idle.
import torch
from torch.profiler import profile, ProfilerActivity

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
             record_shapes=True) as prof:
    for _ in range(8):
        train_one_step()                 # forward, backward, all-reduce, optimizer
        prof.step()

# Key averages expose time spent in NCCL all-reduce vs in matmul kernels;
# (compute time) / (compute + comm + idle) is the measured single-step efficiency.
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))

5. Project Ideas Intermediate

The exercises below ask for short calculations and code edits; these projects are larger, open-ended investigations that turn this chapter's models into measurements on a real system. Each is a good fit for a course assignment or a self-directed weekend.

• Measure the scaling curve of a real data-parallel job. Take any small model and train it with PyTorch DistributedDataParallel on 1, 2, 4, and 8 GPUs (a single multi-GPU machine or a cloud instance suffices). Record wall-clock per epoch, compute the measured speedup and efficiency $E(K)$, and overlay your points on the Amdahl-plus-communication model of Code 3.9.1 by fitting $s$ and $c$. Report where your measured cost curve bottoms out and how far that is from the fastest configuration. The evaluation discipline for trustworthy measurements like these is the subject of Chapter 5.
• Build a cost-optimal node-count recommender. Generalize Code 3.9.1 into a small tool that takes a measured efficiency curve, a node price, and a fixed overhead rate, and returns $K^{\star}$ plus a plot of cost versus $K$. Extend it with a spot-instance mode: model spot price as a fraction of on-demand with a preemption probability, and show how cheaper-but-flaky nodes move $K^{\star}$, a direct preview of the cost-aware scheduling in Chapter 33.
• Audit a serving fleet for cost-per-token. Using a local inference server (or a simulator), drive a request stream at increasing replica counts, measure tokens per second and the resulting cost per million tokens, and find the replica count that minimizes unit cost under a latency constraint. Compare the cost-optimal fleet size against the throughput-maximal one and quantify the gap, the serving analogue of $K^{\star}$ developed further in Chapter 24.