Benchmarking Methodology and Pitfalls

"They asked me how fast I was. I gave them my best lap, on a cold track, with the other tenants asleep, and called it the speedup. The graph was beautiful and the number was a lie."

A Benchmark That Peaked Too Soon

Every later chapter that claims a parallel method is faster, cheaper, or more scalable rests on a measurement, and a measurement you cannot trust is worse than none, because it travels into slides, papers, and capacity plans wearing the authority of a number. Distribution makes the measurement problem harder in three specific ways. The workload now spans many machines, so "the time" is the time of the slowest one, not the average. The environment is shared, so other tenants, thermal throttling, and network contention inject variance that a single-machine benchmark never sees. And the comparison is structurally tempting to rig, because a distributed system is almost always being sold against a single-node baseline, and the easiest way to look good is to leave that baseline unoptimized. We take the methodology first, then the pitfalls, then the standard suites that exist precisely to make cross-system numbers comparable.

1. Define the Workload and the Metric Before You Measure Beginner

The first discipline is to write down, before running anything, exactly what you will run and exactly what you will report. The workload specification fixes the model, the input distribution, the batch size, the sequence length, the numerical precision, the number of machines, and the interconnect. The metric specification fixes the unit (tokens per second, images per second, end-to-end request latency), the aggregation (median, mean, a tail percentile), and the constraint the measurement runs under (for example, "throughput at or below a 200 millisecond p99 latency"). This sounds bureaucratic until you notice that almost every misleading benchmark is misleading because one of these was left floating and silently chosen, after the fact, to flatter the result.

Fixing the metric first also forces you to choose between throughput and latency as the headline, which for distributed serving is a genuine decision rather than a formality. A system can be tuned to maximize tokens per second by batching aggressively, which raises tail latency, or tuned to minimize latency by batching little, which lowers throughput. The two live on a trade-off curve, developed for serving fleets in Chapter 23, and a number reported without its position on that curve is not interpretable. The honest unit for a latency-constrained service is goodput, the throughput achieved while still meeting the latency target, which we defined in Section 5.3 and return to as a pitfall in Section 4 below.

2. Warm Up, Then Measure Steady State Across Many Trials Beginner

The first time a workload runs, it pays costs it will never pay again: just-in-time compilation of kernels, the first population of caches, lazy allocation of buffers, a cold key-value cache, autotuning of a convolution or a collective algorithm. Including that first iteration in the reported number measures the setup, not the steady-state behavior you actually care about, and the inflation can dwarf the effect you are studying. The remedy is to run a few warmup iterations and discard them, then measure many timed iterations once the per-iteration time has settled, exactly the timeline in Figure 5.6.1.

One timed iteration is never enough, because a distributed system's per-iteration time is a random variable, not a constant. Run a fixed number of trials and report a central value with its spread. Prefer the median over the mean: the median resists the occasional slow iteration caused by a garbage-collection pause, a network hiccup, or a noisy neighbor, while the mean is dragged toward those outliers. Never report best-of-N, the fastest single trial, because that is an estimate of the luckiest moment rather than the typical one, and it improves automatically the more trials you run, which makes it meaningless as a basis for comparison. The code below makes all three effects visible on one small workload.

import time, statistics

# A workload with a real ONE-TIME setup cost, the everyday shape of a
# benchmark target: the first call builds a cache / compiles / allocates a
# large buffer; every later call reuses it. This stands in for JIT compilation,
# a cold KV cache, or a lazily built index in a distributed serving path.
_cache = {}

def workload(x):
    if "table" not in _cache:                 # paid exactly once, on the cold call
        time.sleep(0.040)                     # 40 ms of one-time setup
        _cache["table"] = [i * i for i in range(50_000)]
    total = 0
    for v in _cache["table"][:20_000]:        # steady-state work, every call
        total += (v ^ x) & 0xFF
    return total

def timed(x):
    t0 = time.perf_counter()
    workload(x)
    return (time.perf_counter() - t0) * 1e3   # milliseconds

# Measure 25 trials with NO warmup: the cold first call is included.
raw = [timed(i) for i in range(25)]

cold_first   = raw[0]
mean_all     = statistics.mean(raw)            # naive: includes the cold call
mean_warm    = statistics.mean(raw[5:])        # drop 5 warmups, steady state
median_warm  = statistics.median(raw[5:])
best_warm    = min(raw[5:])
stdev_warm   = statistics.pstdev(raw[5:])

print(f"cold first iteration          : {cold_first:7.3f} ms")
print(f"mean over ALL trials          : {mean_all:7.3f} ms   <- inflated by the cold call")
print(f"mean after 5 warmups dropped  : {mean_warm:7.3f} ms")
print(f"median after 5 warmups        : {median_warm:7.3f} ms   <- report this")
print(f"best-of-N after warmup        : {best_warm:7.3f} ms   <- optimistic, do not report")
print(f"stdev after warmup            : {stdev_warm:7.3f} ms")
print(f"cold inflates the mean by     : {(mean_all/median_warm - 1)*100:6.1f} %")
print(f"best-of-N understates by      : {(1 - best_warm/median_warm)*100:6.1f} %")

cold first iteration          :  46.517 ms
mean over ALL trials          :   3.118 ms   <- inflated by the cold call
mean after 5 warmups dropped  :   1.353 ms
median after 5 warmups        :   1.152 ms   <- report this
best-of-N after warmup        :   0.951 ms   <- optimistic, do not report
stdev after warmup            :   0.481 ms
cold inflates the mean by     :  170.7 %
best-of-N understates by      :   17.5 %

The lesson generalizes directly to clusters. On a single machine the cold cost is compilation or a cold cache; on a cluster it is also the first collective call autotuning its algorithm, the first all-gather paging weights in, and the gradual filling of the network's flow tables. The numbers in Output 5.6.1 came from one process, but the shape of the bias, a tall cold iteration dragging the mean upward, is exactly what you see when the first training step of a distributed run takes seconds longer than the thousandth. The reproducible-measurement machinery for multi-node runs, including how to synchronize timers across machines so "the time" is the slowest worker's time, is the subject of Section 5.7.

import torch
from torch.utils.benchmark import Timer

# Times a real GPU kernel correctly: handles warmup, runs enough iterations to
# beat timer noise, and SYNCHRONIZES so you measure compute, not just the launch.
x = torch.randn(4096, 4096, device="cuda")
t = Timer(stmt="x @ x", globals={"x": x})
measurement = t.blocked_autorange(min_run_time=1.0)   # auto-picks trial count
print(measurement.median * 1e3, "ms median")          # reports a robust statistic

3. Control for the Noise the Cluster Injects Intermediate

A clean steady-state median is still wrong if the environment moved underneath it. Distributed systems run on shared hardware, and three sources of noise routinely shift results by more than the effect under test. The first is other tenants: a co-located job saturating the network or the memory bus steals bandwidth your benchmark assumed it owned. The second is thermal behavior: a sustained workload heats the accelerators until they throttle their clocks, so a thirty-second run and a thirty-minute run of the same code report different throughputs, and only the longer one reflects production. The third is placement: on one launch your workers land on nodes sharing a fast NVLink island, on the next they are scattered across slower links, and the all-reduce cost, modeled in Chapter 3 and measured in Chapter 4, changes with the placement even though the code did not.

Controlling for this noise means pinning what you can and reporting what you cannot. Pin the placement (request the same node set, the same topology), run long enough to reach thermal steady state, and where possible isolate the benchmark from other tenants. For the variance you cannot eliminate, run multiple independent trials across different launches, not just multiple iterations within one launch, and report the spread across launches; that spread is the honest uncertainty of your number. A result reported as a single value with no variance is making an implicit claim of zero noise that no shared cluster can support.

4. The Pitfalls That Turn Measurements Into Marketing Intermediate

With clean methodology in hand, the remaining failures are comparisons that are unfair by construction. Table 5.6.1 catalogs the recurring ones, each of which produces a number that is individually defensible but collectively misleading, and each of which has a one-line fix. The unifying theme is construct matching: a comparison is valid only when the two numbers differ in exactly the one variable you claim to be studying and are identical in everything else.

The weak baseline is the most common and the most consequential, because it is the easiest to commit without noticing. A distributed training method that runs 6x faster than a single GPU may run only 1.5x faster than the same GPU using mixed precision, a larger batch, and a compiled kernel, all of which are per-node optimizations cataloged in Chapter 22. The 6x figure is not false, but it credits distribution for speedups that single-node tuning would have delivered for free. The discipline is to make the single-node baseline as fast as you genuinely can before you measure what distribution adds on top, so the comparison isolates the contribution of scaling out from the contribution of basic engineering.

The construct-mismatch pitfall is subtler and shows up constantly in distributed-AI papers. Suppose you report that your system reaches a target accuracy in fewer GPU-hours than a baseline, but the accuracy came from one run on one data split and the GPU-hours came from a different run on a different cluster with a different batch size. Each number is real; the comparison is invalid because the two numbers describe different experiments. A number-by-number audit can confirm each figure is backed by some log and still bless an invalid claim. The only defense is to co-compute the metrics you intend to compare in a single pass, on a single configuration, with the same seed and split, and save them together as one artifact, so that "is each number backed?" and "is the comparison valid?" become the same question.

5. Standard Suites Make Cross-System Numbers Comparable Intermediate

Everything above is what you must do to make your own numbers trustworthy. Standard benchmark suites exist to solve the harder problem of making different organizations' numbers comparable, by fixing the workload, the quality target, and the measurement rules so the only remaining variable is the system under test. MLPerf, run by the MLCommons consortium, is the most widely used in this space. MLPerf Training fixes a model and a dataset and measures the wall-clock time to train to a defined target accuracy, which closes the apples-to-oranges loophole by forbidding the comparison of systems that reached different quality. MLPerf Inference fixes the model and measures throughput and latency under defined query-arrival scenarios (single-stream, server, offline), with the server scenario enforcing a latency bound so that reported throughput is goodput rather than SLO-violating throughput.

The value of a suite like MLPerf is less the leaderboard than the rulebook: a closed division that pins the model and hyperparameters so results are strictly comparable, an open division that allows innovation but labels it as such, and an auditing process that catches the cherry-picking and construct-mismatch errors of Table 5.6.1 before publication. You will not always run a full MLPerf submission, but its rules are a checklist for any serious benchmark of your own, and the evaluation patterns for the serving and retrieval systems you meet later, distributed inference in Chapter 23 and distributed retrieval in Chapter 25, inherit the same definitions of goodput and tail latency that MLPerf Inference codified.

We now have the two halves of honest benchmarking: a methodology (fix the workload and metric, warm up, measure steady state across many trials, report the median with its spread, control for cluster noise) and a catalog of the pitfalls (weak baselines, apples-to-oranges points, cherry-picking, tail-latency blindness, silent SLO violations, and construct mismatch) that misleading numbers exploit. What remains is to make a measurement not merely honest but reproducible by a third party on a different cluster, which requires controlling versions, seeds, and the cross-machine timing that single-node benchmarking never confronts. That is the work of Section 5.7.