Reproducible Measurement on Clusters

"My benchmark was perfectly reproducible. It reproduced a different number every single time, with admirable consistency."

An All-Reduce That Has Seen Some Gradients

The previous sections of this chapter built the vocabulary of evaluation: throughput and latency, scaling efficiency, cost per unit of useful work, and the statistics needed to tell a real improvement from noise. All of it assumes the underlying measurement can be trusted, which in turn assumes it can be reproduced. On one machine that assumption is cheap. On a cluster it is the single hardest property to maintain, because distribution introduces sources of variation that have no single-machine analogue. Before we can certify any of Chapter 5's numbers, we have to confront why two runs of the same distributed job disagree, and what it takes to make them agree, or at least to make their disagreement small, bounded, and documented.

1. Why Two Runs of the Same Cluster Job Disagree Beginner

Single-machine nonreproducibility has familiar, controllable causes: an unset random seed, an uncontrolled thread count, a dependency that silently upgraded. The cluster adds five sources that are structural, not accidental, and each one resists the seed-and-rerun reflex that works on a laptop. Naming them is the first half of the cure, because a source of variation you have not named is a source you cannot pin or report.

The first is nondeterministic reductions. A distributed sum, the all-reduce at the heart of data-parallel training (Chapter 4), combines one partial result per worker. Floating-point addition is not associative, so the order in which those partials are combined changes the last bits of the answer, and that order depends on which worker's bytes arrive first, which is a property of the network on that run, not of your code. The second is variable network conditions: bandwidth and latency between nodes fluctuate, so the wall-clock time of every communication step, and therefore your throughput number, moves between runs even when the computed answer is fixed. The third is multi-tenant interference: on a shared cluster, other jobs contend for the same network links, memory bandwidth, and storage, and their load is invisible to you and uncorrelated with your schedule. The fourth is autoscaling: an elastic job may run on eight workers one day and twelve the next, changing both the timing and, through the reduction order, the arithmetic. The fifth is hardware heterogeneity: a pool labeled with one accelerator name may contain several silicon revisions, firmware versions, and thermal states, each with slightly different numerics and markedly different speed.

2. Collective Nondeterminism, Demonstrated Intermediate

The most surprising of the five sources, because it touches the answer and not merely the timing, is the nondeterministic reduction. It is worth seeing in its smallest possible form, with no cluster and no network, because the mechanism is pure arithmetic: summing the same numbers in a different order gives a different result. On a cluster, the order is chosen for you by message-arrival timing, so the effect appears whether or not you asked for it. The code below sums one million floating-point values, deliberately mixing large and tiny magnitudes so that rounding bites, in four different orders that stand in for four different worker-arrival orderings.

import numpy as np

# Simulate a floating-point all-reduce: the SAME partial gradients, summed in
# different orders, as when workers arrive in a different order on each run.
rng = np.random.default_rng(7)
P = 1_000_000
vals = rng.standard_normal(P).astype(np.float64)
vals[::3] *= 1e8                                  # mix magnitudes so rounding bites

def ordered_sum(x):
    s = 0.0
    for v in x:                                   # left-to-right, like a real reducer
        s += v
    return s

ascending  = ordered_sum(np.sort(vals))           # four worker-arrival orderings
descending = ordered_sum(np.sort(vals)[::-1])
shuffled1  = ordered_sum(rng.permutation(vals))
shuffled2  = ordered_sum(rng.permutation(vals))
exact      = float(np.sum(vals, dtype=np.longdouble))   # higher-precision reference

orders = [ascending, descending, shuffled1, shuffled2]
print("sum, ascending order    :", repr(ascending))
print("sum, descending order   :", repr(descending))
print("sum, worker order A     :", repr(shuffled1))
print("sum, worker order B     :", repr(shuffled2))
print("max spread across orders:", f"{max(orders) - min(orders):.6e}")
print("relative spread         :", f"{(max(orders) - min(orders)) / abs(exact):.2e}")

sum, ascending order    : np.float64(-20189323766.140797)
sum, descending order   : np.float64(-20189323766.132816)
sum, worker order A     : np.float64(-20189323766.296597)
sum, worker order B     : np.float64(-20189323766.29571)
max spread across orders: 1.637802e-01
relative spread         : 8.11e-12

The disagreement is tiny, around eight parts in a trillion here, but it is not zero, and it is not noise you can seed away: each ordering is perfectly deterministic on its own, yet the cluster picks a different ordering each run. The right response is rarely to chase bit-exactness, which on many accelerators costs real speed and is sometimes impossible across hardware revisions. The right response is to report the effect: state that the reduction is nondeterministic, quantify the run-to-run spread by repeating the run, and let that spread set the resolution below which two results are indistinguishable. A measured difference smaller than the reduction-order spread is not an improvement; it is the arithmetic breathing. Relating this $O(10^{-12})$ floating-point spread to the statistical-significance machinery of Section 5.6 is exactly how you decide whether a reported gain clears the noise floor.

3. Practices That Make a Cluster Measurement Reproducible Intermediate

Reproducibility on a cluster is the product of a handful of disciplined habits, each aimed at one of the five sources from Section 1. None is exotic; the discipline is in doing all of them, every run, and recording the result where a stranger can find it.

Pin software and container versions. The single largest source of "it worked last month" is a dependency that moved: a new CUDA driver, a patched collective-communication library, a framework minor release that changed a default. Ship the job as a container image referenced by digest, not by a mutable tag, so the exact bytes of every library are frozen. Fix seeds where possible, and acknowledge what they cannot fix. Set the seeds for every random number generator (the framework's, the data loader's, the language runtime's) and enable the framework's deterministic-algorithm mode where the workload tolerates its speed cost. Then state plainly that collective reductions remain nondeterministic, as Output 5.7.1 showed, so a reader does not expect bit-exactness you cannot deliver. Record the full environment. Capture the hardware (accelerator model and count, silicon revision), the interconnect topology, every library version, and the collective-library settings (for example the NCCL environment variables that select algorithm and protocol), because these change both numerics and speed. Report configuration alongside every number. A throughput figure without its worker count, batch size, precision, and the flags in force is uninterpretable; the configuration is part of the measurement, not metadata about it. Publish a reproducibility package. Bundle the code commit, the data version, the environment description, the configuration, and the seeds into one artifact, as Figure 5.7.1 lays out, so reproduction is a download and a single command rather than an archaeology project.

import os, json, subprocess, random, numpy as np, torch

def make_reproducible(seed: int):
    random.seed(seed); np.random.seed(seed)
    torch.manual_seed(seed); torch.cuda.manual_seed_all(seed)
    torch.use_deterministic_algorithms(True, warn_only=True)  # numeric repeatability
    os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")

def capture_environment(path="env.json"):
    env = {
        "git_commit": subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip(),
        "torch": torch.__version__,
        "cuda": torch.version.cuda,
        "world_size": int(os.environ.get("WORLD_SIZE", 1)),
        "gpu": torch.cuda.get_device_name(0) if torch.cuda.is_available() else "cpu",
        "nccl_env": {k: v for k, v in os.environ.items() if k.startswith("NCCL_")},
    }
    with open(path, "w") as f:
        json.dump(env, f, indent=2)          # ships inside the reproducibility package
    return env

make_reproducible(20260616)
env = capture_environment()

4. From a Single Number to a Reproducible Package Advanced

The practices of Section 3 converge on one deliverable, drawn in Figure 5.7.1: a reproducibility package that bundles code version, data version, environment, configuration, and seeds so a second party can recover the result. Two of those five inputs reach beyond this chapter and tie reproducible measurement to the rest of the book. The data version is only meaningful if the dataset itself is versioned and addressable, which is the job of distributed storage and data loading; pinning "the training set" to an immutable snapshot or content hash, rather than a moving directory, is what we build in Chapter 8. The configuration and environment records are only useful if they are tracked, searchable, and attached to every run automatically rather than by hand, which is the province of experiment tracking and the broader operational discipline developed in Chapter 26. A reproducibility package, in other words, is not a one-off zip file; it is the point where the data-versioning of Part II and the experiment-tracking of Part V meet the evaluation discipline of Part I.

With the package in hand, the loop of Chapter 5 closes. You measure throughput, latency, scaling efficiency, and cost; you establish with the statistics of Section 5.6 that a difference is real; and you seal the whole apparatus into an artifact that lets a skeptic, including your future self, recover the number and the conditions that produced it. That is the difference between a system you can report on and a system you merely ran.

Where Part II begins. Part I established that AI at scale is a distributed system and gave the foundation to reason about it: the axes of distribution, the systems concepts, the performance models, the communication primitives, and the evaluation discipline you just sealed into a reproducible package. The recurring pressure that opened the book was that the data itself outgrew one machine first, before models and before request volume. Part II takes up that pressure directly. It builds the engines of distributed data processing for AI, starting with the model that made web-scale computation routine, MapReduce, whose shuffle is the distant ancestor of the all-reduce you have been studying. We begin in Chapter 6.