Failure Is the Norm at Thousand-GPU Scale

"They asked me to run for two months without stopping. I lasted ninety minutes, twice an hour, on a good day."

A Spot Instance, Moments Before Preemption

The previous three chapters treated a training job as a fixed set of workers that start together, run together, and finish together. That picture is correct for an afternoon on eight GPUs. It quietly stops being correct as the GPU count and the wall-clock both grow, and the reason is not subtle. Each accelerator, each network link, each power supply, and each cooling loop has some small probability of failing on any given day. One machine, observed for a day, almost never fails. Thousands of machines, observed for two months, fail constantly. The job does not get to choose whether failures happen; it only gets to choose whether a failure ends the run or merely dents it. The discipline of making that choice the second way, keeping a long job alive through a steady drizzle of hardware faults, is elastic and fault-tolerant training, and it begins by taking the failure rate seriously as a number rather than a nuisance.

We built the conceptual vocabulary for this in Section 2.4, where failure detection, recovery, and the distinction between a crashed process and a slow one were introduced as general distributed-systems concerns. Here we specialize those ideas to the one workload that stresses them hardest: a single tightly coupled computation, running for months, that cannot make progress unless almost every participant is healthy at almost every step.

1. The Arithmetic of Unreliability Intermediate

Start with one number you can defend and follow it to a conclusion you cannot escape. Suppose a single GPU, together with the host, network, and power that keep it useful, has a probability $p$ of suffering a job-ending fault on any given day. A reasonable operational figure is a fraction of a percent; we will use $p = 0.005$, half a percent per GPU per day. For one GPU this is reassuring: across a two-month run it will almost certainly survive. The danger appears only when the GPUs are many and they are coupled, because a synchronous job fails when any of its members fails.

Model each GPU's faults as a Poisson process, the standard assumption for independent rare events. A per-day fault probability $p$ corresponds to a fault rate of $\lambda_{\text{gpu}} = -\ln(1 - p)$ faults per GPU per day. Independent rates add, so a job of $G$ GPUs fails at rate $\lambda_{\text{job}} = G\,\lambda_{\text{gpu}}$. The mean time between job failures, the job-level MTBF, is the reciprocal of that combined rate,

The shape of these two expressions is the whole story. The MTBF shrinks like $1/G$: double the GPUs and you halve the time between interruptions. The expected number of failures over a fixed run grows like $G$: every GPU you add is one more thing that can take down the entire job. There is no regime in which adding workers makes a synchronous job more reliable; it can only make the wall-clock shorter, and it does so while making interruptions more frequent. The code below evaluates both formulas across a realistic range of cluster sizes so the trend is concrete rather than asymptotic.

import math

# Each GPU (with its host, network, and power) has a small per-day fault probability.
p_fail_per_gpu_day = 0.005          # 0.5% chance one GPU faults on a given day
run_days = 60.0                     # a two-month training run
hours_per_day = 24.0

def job_mtbf_hours(num_gpus, p_day):
    rate_per_gpu_day = -math.log(1.0 - p_day)        # Poisson rate from per-day prob
    rate_job_hour = num_gpus * rate_per_gpu_day / hours_per_day   # rates add across GPUs
    return 1.0 / rate_job_hour                       # mean time between job failures

def expected_failures(num_gpus, p_day, days):
    return num_gpus * (-math.log(1.0 - p_day)) * days

print(f"{'GPUs':>8} {'job MTBF (h)':>14} {'exp. failures':>16}")
for n in [1, 8, 64, 512, 4096, 16384]:
    print(f"{n:>8} {job_mtbf_hours(n, p_fail_per_gpu_day):>14.1f}"
          f" {expected_failures(n, p_fail_per_gpu_day, run_days):>16.1f}")

    GPUs   job MTBF (h)    exp. failures
       1         4788.0              0.3
       8          598.5              2.4
      64           74.8             19.2
     512            9.4            154.0
    4096            1.2           1231.9
   16384            0.3           4927.5

Read Output 18.1.1 next to Figure 18.1.1 and the lesson lands. The single-GPU column is the comforting world the previous chapters assumed: a fault is a once-in-a-blue-moon event you can ignore. By the time the job spans four thousand accelerators, the MTBF has fallen below a working lunch and the run will be interrupted more than a thousand times. Nothing about the hardware degraded; the arithmetic of many independent parts simply caught up with the wall-clock. A training system designed as if failures were rare is, at this scale, a system designed to fail.

2. Why Synchronous Parallelism Makes One Failure Fatal Intermediate

The data parallelism of Chapter 15, the sharded parallelism of Chapter 16, and the expert parallelism of Chapter 17 share a structural feature that is the source of their fragility. Every training step ends in a collective: an all-reduce of gradients, a reduce-scatter and all-gather of sharded parameters, an all-to-all of expert activations. A collective is a barrier. It cannot complete until every participant has arrived with its contribution, because the result each worker receives is a function of all the inputs. If one worker is gone, the collective never returns; the surviving workers block on it forever, holding their GPUs idle while they wait for a peer that will never speak again.

This is why a single failure is not a local event but a global one. There is no notion of the job limping along at reduced capacity, the way a web service drops one replica and serves the rest. A synchronous training step is all-or-nothing: either all workers reach the collective and the step succeeds, or one does not and the step, and every step after it, hangs. Without intervention, the practical outcome is that the whole job is declared dead, every worker is torn down, and the run restarts. The very property that made distributed training exact, the collective that combines all contributions into one coherent update, is also what makes it brittle. We are not trying to abandon synchronous training; its convergence behavior is too good. We are trying to wrap it in machinery that detects the missing participant, removes it, and lets the survivors continue, which is precisely the elastic-membership problem of Section 18.4.

3. The Ruinous Cost of Restart-From-Scratch Intermediate

Suppose the job has no fault tolerance beyond "if anything dies, relaunch the whole thing from the beginning." Each interruption then discards every GPU-hour spent since the run started. Combine that with Output 18.1.1, where a 4096-GPU run expects more than a thousand interruptions, and the waste compounds catastrophically: the run can spend the overwhelming majority of its compute repeatedly re-doing work it had already done, never accumulating enough uninterrupted progress to finish. The alternative is to write the training state to durable storage periodically, a checkpoint, so that a failure rewinds the job only to the last checkpoint rather than to day zero. The code below contrasts the two policies in GPU-hours, the currency that turns into dollars on the cloud bill.

import math

p, run_days, hpd = 0.005, 60.0, 24.0
G = 4096
ef = G * (-math.log(1.0 - p)) * run_days        # expected interruptions over the run
ckpt_interval_h = 1.0                            # checkpoint every hour

# Restart-from-scratch: each failure discards everything since the run began.
# Bound the expected lost wall-clock per failure by the mean elapsed time, run/2.
naive_lost_gpu_h = ef * (run_days * hpd / 2.0) * G

# Checkpointing: a failure rewinds only to the last checkpoint, losing on
# average half a checkpoint interval of wall-clock.
ckpt_lost_gpu_h  = ef * (ckpt_interval_h / 2.0) * G

print(f"expected interruptions   : {ef:.1f}")
print(f"restart-from-scratch lost: {naive_lost_gpu_h:,.0f} GPU-hours")
print(f"checkpoint-every-1h lost : {ckpt_lost_gpu_h:,.0f} GPU-hours")
print(f"reduction factor         : {naive_lost_gpu_h / ckpt_lost_gpu_h:,.0f}x")

expected interruptions   : 1231.9
restart-from-scratch lost: 3,632,968,665 GPU-hours
checkpoint-every-1h lost : 2,522,895 GPU-hours
reduction factor         : 1,440x

The restart-from-scratch number in Output 18.1.2 is deliberately absurd; it is larger than the total compute any organization would ever spend, and that is exactly why no frontier run is operated that way. The comparison makes the abstract argument financial. At a representative cloud price of a few dollars per GPU-hour, the gap between the two policies is the difference between a feasible project and an impossible one. Checkpointing does not eliminate the cost of failure, since each interruption still loses some work and some recovery time, but it converts an unbounded loss into a small, predictable tax. The next two sections turn that tax down further by making the checkpoints themselves fast and cheap to write.

4. What the Frontier Reports Beginner

The arithmetic above is not a worst-case fantasy; it matches what operators of the largest published runs describe. Public training reports for frontier models read, in their reliability sections, like field notes on a constant stream of hardware faults, and the engineering they describe is overwhelmingly about surviving those faults rather than computing faster. The pattern is consistent across organizations and model families: at the thousand-to-tens-of-thousands GPU scale, interruptions are measured in events per day, and the systems are built to absorb them.

# Elastic launch: run with anywhere from 4 to 8 healthy nodes, 8 GPUs each.
# A node failure shrinks the group and restarts from the last checkpoint,
# instead of hanging the whole job on a dead all-reduce.
torchrun \
  --nnodes=4:8 \
  --nproc_per_node=8 \
  --max_restarts=50 \
  --rdzv_backend=c10d \
  --rdzv_endpoint="$HEAD_NODE:29500" \
  train.py

5. The Shape of the Chapter Beginner

The two formulas of Section 1 define every problem this chapter solves, because each piece of machinery is an attack on one term in the cost of an interruption. The cost of a single interruption is, roughly, the work lost since the last save plus the time to detect the failure plus the time to recover and resume. Lowering the first term is checkpointing, made fast and non-blocking in Section 18.2 and made correct on resume, with deterministic replay, in Section 18.3. Keeping the job alive across the membership change, rather than tearing it down, is elasticity in Section 18.4. Two failure modes that are not outright crashes get their own treatment: the worker that is alive but slow, handled by straggler detection in Section 18.5, and the worker that is healthy but reclaimed by the cloud, handled by preemption and spot-instance training in Section 18.6. Section 18.7 adds memory offload across the hierarchy so that a job survives not only lost workers but exhausted device memory, and Section 18.8 closes the loop with the observability needed to see all of this happening in time to react.

With the arithmetic of unreliability established, the per-failure cost named term by term, and the chapter mapped onto those terms, the natural first move is to attack the largest and most controllable term: the work lost since the last save. That means writing checkpoints, and writing them fast enough that the saving does not itself become the bottleneck. Section 18.2 takes up distributed, sharded, asynchronous checkpointing.