Why Distribution Needs Its Own Evaluation Discipline

"They asked if I was faster on sixteen machines. I said yes. Nobody asked if I was cheaper, or correct, or still alive at midnight. I was none of those things."

A Benchmark That Reported Only the Number It Liked

For a model trained and served on one machine, evaluation has a comfortable center of gravity: a held-out accuracy, an F1, a perplexity. You report that number, perhaps a confidence interval around it, and a reader knows roughly what they are getting. The number is portable because the machine is invisible; nothing about the result depends on how many cores were busy or how the bytes moved. That comfort is exactly what distribution takes away. Once the data is sharded, the gradient is summed over a network, the model is split across devices, and the service answers from a fleet, the question "is this system good?" fractures into several questions that a single accuracy number cannot hold. This section is about why that fracture happens and what replaces the single number: a small set of metrics, measured together, that together describe a distributed system faithfully.

1. Why One Accuracy Number Is Not Enough Beginner

Imagine two teams that each built the same recommendation model and each report 0.91 AUC on the same test set. By the single-machine standard they are tied. Now add the facts that distribution exposes. The first team reaches that AUC on four machines in two hours at a rental cost of forty dollars, with a serving fleet that holds its latency target when one replica dies. The second reaches the same AUC on sixteen machines in ninety minutes at three hundred dollars, with a serving path that drops a tenth of its requests whenever a node restarts. The accuracy number says they are equal. Everything that matters for running the system in production says they are not. The single number was never wrong; it was simply blind to the dimensions that distribution makes real.

The blindness has a structure worth naming. A single-machine accuracy assumes the computation that produced it is fixed and reliable, so the only variable is the model's quality. Distribution makes the computation itself a variable: it can be faster or slower depending on how many machines and how they talk; it can be cheaper or more expensive depending on utilization; it can succeed or partially fail depending on what broke during the run. A verdict that reports only quality, while staying silent on speed, price, and survival, is answering a question nobody running a cluster is actually asking.

2. Correctness: Did Distribution Preserve the Result? Intermediate

The first thing distribution can break is the answer itself, and this is the dimension most likely to be assumed away. Section 1.1 proved that data-parallel training can be exact: the gradient is an average over examples, an average decomposes over shards, and summing the per-shard partial sums and dividing by $N$ recovers the identical single-machine gradient,

where $S_1, \dots, S_K$ partition the $N$ examples across $K$ workers. That identity is the gold standard a correctness check measures against: a distributed training run should reproduce the single-machine result up to floating-point rounding, and when it does not, the gap is a bug to be hunted, not a tolerance to be accepted. But the identity holds only when the implementation honors it. Unequal shard sizes combined with a plain unweighted average, non-deterministic reduction orders that change the rounding, a dropped worker whose shard silently vanishes from the sum, or a mismatched random seed across replicas all produce a result that differs from the single-machine answer in ways that no accuracy number on a single test set will reveal. Correctness evaluation for a distributed system therefore means more than "did it score well"; it means "did distribution change the computation it was supposed to preserve." We make this concrete by always reporting the distributed result alongside a single-machine or smaller-scale reference whenever one can be computed.

3. Performance, Cost, and Reliability: The Three New Axes Beginner

Beyond correctness, distribution adds three axes that simply do not exist on one machine. Performance asks whether adding machines bought proportional speed: a system on eight machines that runs only 2.6 times faster than on one has spent five machines' worth of hardware to buy 1.6 machines' worth of speed. The metrics that capture this, speedup and efficiency, are the subject of Section 5.2, and the communication overhead that erodes them, the very tax introduced in Chapter 3, gets its own ratio in Section 5.4. Cost asks whether the speed was worth its price: more machines for a shorter time can cost more in total even when it finishes sooner, and energy is increasingly the binding currency, so Section 5.5 develops cost-per-step, utilization, and energy accounting. Reliability asks whether the system survives the failures a large cluster makes routine: at a thousand machines something is always broken, so throughput under failure (goodput) and tail latency under partial degradation, taken up in Section 5.3, become part of the verdict rather than an afterthought.

These three axes interact, which is why they must be measured together. Pushing performance by adding machines usually worsens efficiency and raises cost; hardening reliability with replication consumes throughput; chasing the lowest cost can leave a system too lean to absorb a failure. A claim that moves one axis while staying silent on the others is not an evaluation, it is a selected highlight. The next demonstration shows exactly how such a highlight misleads.

4. A Faster Run That Got Worse Intermediate

The most common misleading claim in distributed AI is "we added machines and it got faster," stated as if speed were the whole story. It is true and almost meaningless on its own. The code below takes measured per-step wall-clock times for a training job at one, two, four, and eight nodes, where the speedup is sublinear because communication grows with the node count, and computes the three numbers that "it got faster" hides: speedup relative to one node, efficiency (speedup divided by node count), and cost per step in dollars (node count times step time times a rental rate).

base_time = 10.0          # seconds per training step on 1 node
cost_per_node_sec = 0.02  # dollars per node-second (cluster rental)

# Measured wall-clock per step; speedup is sublinear as nodes grow because
# the all-reduce communication per step grows with the worker count.
measured = {1: 10.00, 2: 7.69, 4: 5.00, 8: 3.85}

print(f"{'nodes':>6}{'step_s':>9}{'speedup':>9}{'efficiency':>12}{'cost/step($)':>14}")
for n, t in measured.items():
    speedup = base_time / t
    eff = speedup / n                 # fraction of ideal linear speedup retained
    cost = n * t * cost_per_node_sec  # node-seconds spent per step, priced
    print(f"{n:>6}{t:>9.2f}{speedup:>9.2f}{eff:>11.0%}{cost:>14.4f}")

 nodes   step_s  speedup  efficiency  cost/step($)
     1    10.00     1.00       100%        0.2000
     2     7.69     1.30        65%        0.3076
     4     5.00     2.00        50%        0.4000
     8     3.85     2.60        32%        0.6160

Read the table the way a single-number report would, and the two-node row is a success: 1.30 times faster. Read all four columns and the same row is a warning. Efficiency has dropped to 65%, meaning more than a third of every added machine is being wasted on coordination, and the cost per step has risen from twenty cents to nearly thirty-one, so the faster run is the more expensive run. By eight nodes the picture is stark: a real 2.6x speedup bought at the price of two-thirds of the hardware sitting idle on communication and triple the cost per step. Nothing here is fabricated; the speedup is real. The point is that speedup alone endorsed a configuration that efficiency and cost both condemn. An evaluation that reports only the speedup column is not lying about any single number; it is misleading by omission, and omission is the failure mode this chapter is built to prevent.

5. One Configuration, Co-Computed Metrics Intermediate

The deeper lesson of Output 5.1.1 is not which column to read but where the numbers came from. Every value in that table was computed from one set of measured step times, in one pass, for one configuration. That is the central methodological rule of this chapter, and it is the rule every later part of the book relies on: report construct-matched metrics from a single configuration, co-computed in one run, and never assemble a comparison from numbers that came from different runs. The danger is subtle precisely because it can pass a careless audit. If you quote the speedup from one experiment, the accuracy from a second with a different seed, and the cost from a third on a different cluster, each individual number may be correct, yet the bundle describes no system that ever existed. A reader who checks each figure against its source will find every one "backed," and the comparison will still be invalid, because the numbers are not commensurable.

The remedy is mechanical and cheap: when you measure a distributed run, log the raw quantities (step times, byte counts, request outcomes, joules) for that one configuration, and derive every reported metric from that single log. Speedup, efficiency, cost per step, throughput, and goodput then share a denominator and a provenance; they describe the same machine on the same day doing the same work. This is the construct-matching principle that Section 5.6 turns into a benchmarking methodology and Section 5.7 turns into a reproducible measurement protocol you can run on a real cluster.

6. What This Chapter Builds Beginner

The rest of Chapter 5 develops each dimension of Figure 5.1.1 into a measurable, reproducible practice, and Table 5.1.1 is the map. Read it as the promise of the chapter: by its end you will have a metric and a measurement protocol for every corner of the evaluation vector, plus the methodology to combine them soundly. Each later section assumes the single-configuration rule from Section 5 and the four-dimension framing from this section.

With the four dimensions named and the single-configuration rule in place, the natural next question is how to make the performance dimension precise. What exactly is speedup, how does efficiency fall as machines are added, and what shape does the scalability curve take as the communication tax of Chapter 3 asserts itself? That is where the chapter turns next, in Section 5.2.

import pandas as pd

# One row per configuration; raw measured quantities only.
df = pd.DataFrame({"nodes": [1, 2, 4, 8], "step_s": [10.00, 7.69, 5.00, 3.85]})

base = df.loc[df.nodes == 1, "step_s"].item()
df["speedup"]    = base / df.step_s
df["efficiency"] = df.speedup / df.nodes
df["cost_step"]  = df.nodes * df.step_s * 0.02   # node-seconds, priced
print(df.to_string(index=False))