Distributed Experiment Tracking

"Four hundred runs finished overnight. The one that won is the one nobody wrote down. I am the spreadsheet that was supposed to prevent this."

A Tracking Server With No Record of Its Best Run

The previous section wired up continuous delivery so that a vetted model can flow from commit to fleet automatically. That pipeline assumes you already know which model deserves promotion. Choosing it is the job of experiment tracking. Every serious training effort generates many candidate models: different learning rates, different data mixes, different architectures, each producing a curve of metrics over time. Without a disciplined record, the comparison degrades into screenshots and remembered numbers, and reproducibility, the property we made measurable back in Section 5.7, quietly evaporates. A tracker exists to make every run a first-class, queryable, comparable object.

What turns this from a single-machine logging habit into a distributed-systems problem is scale on two axes at once. A single run is itself distributed across many ranks, so the tracker must decide whose numbers are canonical and how to fold in the rest. A single sweep is hundreds of such runs writing concurrently, so the tracking server is a shared service under real write load. We take these in turn, then build a tracker that handles both.

1. What a Run Record Must Capture Beginner

A run is the atomic unit of experiment tracking, and a useful run record captures six things, each answering a question you will ask later when a result needs to be explained or reproduced. The configuration (the hyperparameters) answers "what knobs were set". The code version (a commit hash) answers "what program ran". The data version (a dataset hash or snapshot id) answers "what did it learn from". The metrics over time (loss, accuracy, throughput per step) answer "how did it go". The environment (library versions, hardware, the launch command) answers "where did it run". The artifacts (checkpoints, plots, the final model) answer "what came out". Drop any one of these and a run that looked decisive becomes a result you can admire but never trust or rebuild.

The tie to reproducibility is direct. Section 5.7 defined a reproducible result as one another team can regenerate from recorded inputs; the six fields above are exactly those inputs, captured automatically at run time rather than reconstructed from memory afterward. The discipline is to log them as the run happens, because the environment that produced a number is hardest to recover once the cluster has moved on to the next job.

2. The Distributed Angle: Who Logs, and What Gets Aggregated Intermediate

A data-parallel training run is not one process emitting metrics; it is $W$ ranks, each computing on its own shard, each capable of logging. If all $W$ ranks log the loss every step, the tracker receives $W$ copies of a quantity that is supposed to be one number, and the leaderboard fills with near-duplicates. The standard resolution is a division of labor. After the gradient all-reduce that synchronizes the model (the collective from Chapter 15), every rank holds the same synchronized weights, so rank 0 logs the canonical metrics that describe the shared model, the validation accuracy, the synchronized loss, the global throughput. The remaining ranks stay silent on those, which keeps the canonical history clean and the write volume down by a factor of $W$.

Per-rank logging does not disappear; it changes purpose. For debugging, you sometimes want every rank's local loss, because a single straggler shard with a corrupted batch or a slow link shows up as a divergence between rank 0's canonical number and the all-rank mean. A tracker that supports both views, rank-0 as the official record and all-rank as a diagnostic overlay, lets you answer both "how is the run doing" and "is any shard misbehaving" from the same store. The canonical metric for a step is the rank-0 value; the all-rank diagnostic is the mean over ranks,

and the gap $\lvert m^{\text{canon}}_{t} - \bar{m}_{t}\rvert$ is a cheap, continuously available skew signal across shards. We will reference this gap in the demo below.

3. The Tracking Server as a Shared Fleet Service Intermediate

The second axis of scale is the number of runs. A hyperparameter search, the subject of Chapter 21, does not run one job; it launches hundreds of concurrent trials, each a distributed run, each streaming metrics. The tracking server is the one component every trial writes to, which makes it a shared fleet service with its own scaling problem. If a sweep of 400 trials, each with 8 ranks logging at a few hertz, all point at one server, the ingest path must absorb thousands of small writes per second without becoming the bottleneck that stalls training.

This is why production trackers separate the cheap, high-frequency write path from the expensive query path. Metrics are appended to a fast log-structured store (often batched and buffered on the client so the network sees one request per hundreds of points), while comparison queries and leaderboards run against an indexed view built asynchronously. The same architectural instinct from the rest of this book applies: the hot path stays simple and append-only, and the coordination-heavy work (ranking, joining across runs) is pushed off the critical path. Figure 26.5.1 shows the shape: many ranks across many runs funnel into one ingest service, which feeds both a comparison dashboard and the model registry from Section 26.3.

4. A Tracker From First Principles Intermediate

The code below is a complete, if tiny, experiment tracker. It implements the central server as an append-only metric store, ingests metrics from multiple simulated ranks across the runs of a hyperparameter sweep, exposes both the rank-0 canonical view and the all-rank diagnostic view, and produces a leaderboard that ranks every run and names the one to promote. Nothing here needs a network or a database; the point is to make the aggregation logic, the part a hosted tracker hides, fully visible.

import math, random
from collections import defaultdict

random.seed(7)

# --- A central, in-memory tracking server the whole "fleet" writes to. ---
class TrackingServer:
    def __init__(self):
        self.runs = {}                       # run_id -> config dict
        self.metrics = defaultdict(list)     # (run_id, rank, key) -> [(step, value)]

    def start_run(self, run_id, config):
        self.runs[run_id] = dict(config)

    def log(self, run_id, rank, step, key, value):   # the high-frequency hot path
        self.metrics[(run_id, rank, key)].append((step, value))

    # rank-0 holds the canonical metric; the all-rank view averages across ranks.
    def series(self, run_id, key, ranks):
        if ranks == "rank0":
            return self.metrics.get((run_id, 0, key), [])
        per_step = defaultdict(list)
        for r in self._ranks_of(run_id):
            for step, val in self.metrics.get((run_id, r, key), []):
                per_step[step].append(val)
        return [(s, sum(v) / len(v)) for s, v in sorted(per_step.items())]

    def _ranks_of(self, run_id):
        return sorted({r for (rid, r, k) in self.metrics if rid == run_id})

    def final(self, run_id, key, ranks="rank0"):
        s = self.series(run_id, key, ranks)
        return s[-1][1] if s else float("nan")


# --- Simulate a sweep: each run is a distributed job of W ranks. ---
def simulate_run(server, run_id, lr, wd, world_size, steps=40):
    server.start_run(run_id, {"lr": lr, "wd": wd, "world_size": world_size})
    floor = 0.18 + 6.0 * (lr - 0.03) ** 2 + 1.5 * wd      # hyperparam-dependent loss floor
    for step in range(steps):
        base = floor + 0.9 * math.exp(-0.12 * step * (lr / 0.03))
        for rank in range(world_size):
            noisy = base + random.gauss(0, 0.02)          # per-rank shard noise
            server.log(run_id, rank, step, "loss", noisy)
        # After all-reduce every rank shares the weights, so rank-0 logs the canonical val_acc.
        server.log(run_id, 0, step, "val_acc", 1.0 - base + random.gauss(0, 0.005))


server = TrackingServer()
sweep = [(lr, wd) for lr in (0.01, 0.03, 0.06) for wd in (0.0, 0.01)]
WORLD = 4
for i, (lr, wd) in enumerate(sweep):
    simulate_run(server, f"run-{i:02d}", lr, wd, WORLD)

print(f"ingested {len(server.runs)} concurrent runs x {WORLD} ranks each")
print(f"total metric points: {sum(len(v) for v in server.metrics.values())}\n")

# Rank-0 vs all-rank view for one run (detecting skew across shards).
rid = "run-02"
r0, allr = server.final(rid, "loss", "rank0"), server.final(rid, "loss", "allrank")
print(f"{rid} final loss  rank-0 (canonical): {r0:.4f}")
print(f"{rid} final loss  all-rank (mean)   : {allr:.4f}")
print(f"{rid} rank skew                      : {abs(r0 - allr):.4f}\n")

# Leaderboard across the sweep, ranked by rank-0 validation accuracy.
board = sorted(
    [(rid, c["lr"], c["wd"], server.final(rid, "val_acc"), server.final(rid, "loss"))
     for rid, c in server.runs.items()],
    key=lambda r: r[3], reverse=True)

print("LEADERBOARD (by rank-0 val_acc)")
print(f"{'run':<8}{'lr':>6}{'wd':>7}{'val_acc':>10}{'loss':>9}")
for rid, lr, wd, acc, loss in board:
    print(f"{rid:<8}{lr:>6}{wd:>7}{acc:>10.4f}{loss:>9.4f}")

best = board[0]
print(f"\nwinning run promoted to registry: {best[0]} "
      f"(lr={best[1]}, wd={best[2]}, val_acc={best[3]:.4f})")

ingested 6 concurrent runs x 4 ranks each
total metric points: 1200

run-02 final loss  rank-0 (canonical): 0.1860
run-02 final loss  all-rank (mean)   : 0.1772
run-02 rank skew                      : 0.0088

LEADERBOARD (by rank-0 val_acc)
run         lr     wd   val_acc     loss
run-04    0.06    0.0    0.8144   0.1510
run-02    0.03    0.0    0.8099   0.1860
run-05    0.06   0.01    0.8036   0.2284
run-03    0.03   0.01    0.7889   0.2133
run-00    0.01    0.0    0.6211   0.3413
run-01    0.01   0.01    0.6144   0.3591

winning run promoted to registry: run-04 (lr=0.06, wd=0.0, val_acc=0.8144)

Three details in that output carry the section's argument. The 1200 points came from six runs writing concurrently to one server, which is the fleet-write pattern in miniature. The rank skew of 0.0088 on run-02 is the all-rank diagnostic doing its job: small here because the shards behave, but the same number would spike if one rank's shard were poisoned, surfacing a bug the rank-0 canonical curve alone would hide. And the leaderboard ranks every run on the same canonical metric, so the promotion decision, run-04 to the registry, is a comparison of like with like rather than of remembered screenshots.

5. Comparing Runs, Leaderboards, and the Link to the Registry Intermediate

The leaderboard in Output 26.5.1 is where tracking pays off. With every run bound to its configuration and ranked on one canonical metric, the sweep becomes a queryable object: filter to runs with weight decay zero, sort by validation accuracy, read off that the higher learning rate wins on this problem. That is the comparison a hosted dashboard renders as a sortable table and a set of overlaid metric curves, and it is the same logic the demo computes by hand. The value is not the rendering; it is that the comparison is fair because the provenance is complete.

The leaderboard's top entry is also the bridge to the rest of the chapter. The winning run carries exactly the provenance the model registry of Section 26.3 needs to register a new model version: the config, the code commit, the data version, and the artifact path. Promotion is then a hand-off of a tracked run, not a fresh upload of a mystery file, which is what makes the CI/CD flow of Section 26.4 trustworthy end to end. The tracked metrics do not stop mattering at promotion either; the validation curves recorded here become the baseline that production monitoring compares against, the connection Section 26.6 builds into fleet-wide observability.

import mlflow                                  # pip install mlflow

with mlflow.start_run(run_name="run-04"):
    mlflow.log_params({"lr": 0.06, "wd": 0.0, "world_size": 8})
    mlflow.set_tag("git_commit", code_version)  # provenance: code + data version
    mlflow.set_tag("data_version", data_hash)
    for step, (loss, val_acc) in enumerate(history):
        if rank == 0:                            # rank-0 logs the canonical metrics
            mlflow.log_metric("loss", loss, step=step)
            mlflow.log_metric("val_acc", val_acc, step=step)
    mlflow.log_artifact("checkpoint.pt")         # the artifact the registry will promote

One operational caveat survives every tool: logging is not free. At high frequency, a synchronous log_metric on the training hot path can stall the step, so trackers buffer points on the client and flush in batches, and you log scalars often but heavy artifacts rarely. The instinct is the same one from Section 3 of this section: keep the hot path append-only and cheap, push everything expensive off it.

We now have the record. Every run that crosses the fleet is captured with its full provenance, aggregated from many ranks into one canonical history, comparable against hundreds of peers on a leaderboard, and linked to the registry that promotes the winner. What that record cannot tell you is how the promoted model behaves once it is serving real traffic across the fleet, where the metrics that matter shift from training loss to latency, throughput, and error rates under load. Turning the tracked training baseline into live fleet-wide monitoring is the subject of Section 26.6.