Throughput, Latency, Cost, and Reliability

"They asked me for my average response time. I gave it to them. They deployed me. Then the ninety-ninth customer of every hundred met the real me."

A Tail Latency Nobody Budgeted For

The previous section sorted AI workloads into batch, streaming, online, and interactive regimes, each with its own sense of what "fast enough" means. We now make "fast enough" precise. A batch training job cares about throughput and total cost and barely notices the latency of any single example; an interactive chatbot lives or dies by the latency of the slowest few percent of requests; a streaming pipeline must sustain a throughput floor or fall permanently behind. The same four metrics describe all of them, but the weighting changes. This section defines each metric formally, connects throughput and latency through the one law every systems engineer should know by heart, shows why the average latency is the wrong thing to optimize once a request fans out across many machines, and explains why a thousand-machine system is a system in which something is almost always broken.

1. Throughput: Work Per Unit Time Beginner

Throughput is the rate at which a system completes useful work. The unit of work depends on the system: training examples per second, tokens generated per second, requests served per second, documents indexed per second. Write it as $X = C / T$, the count $C$ of completed work items divided by the wall-clock time $T$ over which they completed. Throughput is the metric a batch job optimizes above all others, because a training run that finishes in two hours instead of eleven has, for the same correctness, simply done more work per unit time. It is also the metric most people expect to grow linearly when they add machines, and the metric that most often disappoints them, because communication and coordination eat into the gains in ways that Chapter 3 models with Amdahl's law and scaling-efficiency curves.

One refinement matters from the start. Raw throughput counts every completed item, including ones that failed a deadline or returned an error. The quantity that actually serves users is goodput: the rate of work completed correctly and within its latency budget. A serving fleet can report a glorious throughput number while half those responses arrived too late to be useful, in which case its goodput is half its throughput and its dashboard is lying. We treat goodput as a first-class evaluation metric in Chapter 5; for now, hold the distinction that throughput counts attempts and goodput counts wins.

2. Latency and the Tyranny of the Tail Intermediate

Latency is the time a single request takes, measured from when it arrives to when its response is complete. Because no two requests take the same time, latency is not a number but a distribution, and the right way to describe a distribution is by its quantiles. The median, or fiftieth percentile, written p50, is the latency that half of requests beat. The p95 and p99 are the latencies that 95 and 99 percent of requests beat; the slowest one in twenty and one in a hundred fall above them. These upper quantiles are the tail of the distribution, and for any interactive AI system they, not the average, are the numbers that decide whether users are satisfied. Reporting only the mean hides the tail completely: a system with a 40 millisecond average can have a 240 millisecond p99, and it is the p99 that a frustrated user actually experiences several times an hour.

Why does the tail dominate specifically at scale, and not on one machine? Because a distributed request is rarely served by one machine. A single user query to a large model may fan out to dozens of shards (tensor-parallel partitions, retrieval index shards, expert partitions), and the request is not done until the slowest of those parts replies. If a request touches $n$ independent components and each component is fast 99 percent of the time, the probability that the whole request avoids every component's slow tail is $0.99^{n}$. The chance of hitting at least one slow component is

where $p$ is each component's tail probability. With $p = 0.01$ and $n = 100$ fan-out components, that is $1 - 0.99^{100} \approx 0.63$: a request that touches a hundred parts is slow about 63 percent of the time even though each part is slow only 1 percent of the time. This is the fundamental reason tail latency is a distributed-systems problem rather than a single-node one, and why a one-in-a-hundred straggler on any worker can become the common-case experience for whole requests. We confront it directly in distributed serving (Chapter 24), where techniques such as hedged requests and tail-aware scheduling exist precisely to tame this multiplication.

3. Little's Law: The Bridge Between Throughput and Latency Intermediate

Throughput and latency are not independent; they are tied together by the amount of work a system is doing at once. Little's law states that for any stable system, the average number of in-flight requests equals the arrival rate times the average time each request spends in the system:

where $L$ is the average concurrency (requests being worked on simultaneously), $\lambda$ is the throughput (requests arriving and completing per unit time in steady state), and $W$ is the average latency each request experiences end to end. The law is exact, holds for essentially any arrival pattern, and needs no assumption about the latency distribution; it is the closest thing distributed serving has to a conservation law. Its practical force is that you cannot pick all three numbers freely. If you want to triple throughput $\lambda$ at fixed latency $W$, you must hold three times as many requests in flight, which means provisioning three times the concurrent capacity (more replicas, larger batches, more memory for in-flight state). If your concurrency $L$ is capped (a GPU can hold only so large a batch in its KV cache), then pushing $\lambda$ up forces $W$ up: requests queue, and latency, especially the tail, climbs.

The snippet below makes both ideas concrete at once. It simulates the latency of a serving fleet of fast workers plus one slow straggler that takes a fraction of the traffic, computes the p50, p95, and p99 with and without that straggler, and then applies Little's law to convert a target throughput and the measured mean latency into the concurrency the fleet must sustain.

import numpy as np

rng = np.random.default_rng(7)
N = 200_000                                   # number of served requests

# A fleet of identical fast workers plus ONE slow straggler. Base service time
# is lognormal (most requests fast, a few naturally slow); 2% of requests are
# routed to the slow worker and take ~6x longer.
base = rng.lognormal(mean=np.log(40), sigma=0.35, size=N)   # milliseconds
is_slow = rng.random(N) < 0.02                              # 2% hit straggler
lat = np.where(is_slow, base * 6.0, base)

def pct(a):
    p50, p95, p99 = np.percentile(a, [50, 95, 99])
    return p50, p95, p99

p50, p95, p99 = pct(lat)
p50f, p95f, p99f = pct(base)                                # if no straggler

print("WITH one slow worker (2% of traffic)")
print(f"  p50 = {p50:6.1f} ms")
print(f"  p95 = {p95:6.1f} ms")
print(f"  p99 = {p99:6.1f} ms")
print(f"  mean= {lat.mean():6.1f} ms")
print("WITHOUT the straggler (all workers healthy)")
print(f"  p50 = {p50f:6.1f} ms   p95 = {p95f:6.1f} ms   p99 = {p99f:6.1f} ms")
print(f"  p99 inflation from one slow worker: {p99 / p99f:4.2f}x")

# Little's law: L = lambda * W. A served fleet at throughput lambda with mean
# per-request latency W must hold L requests in flight concurrently.
lam = 5000.0                                                # requests / second
W = lat.mean() / 1000.0                                     # seconds
L = lam * W
print()
print("Little's law  L = lambda * W")
print(f"  throughput lambda = {lam:8.0f} req/s")
print(f"  mean latency  W   = {W*1000:8.1f} ms")
print(f"  concurrency   L   = {L:8.1f} requests in flight")

WITH one slow worker (2% of traffic)
  p50 =   40.4 ms
  p95 =   77.0 ms
  p99 =  237.8 ms
  mean=   46.8 ms
WITHOUT the straggler (all workers healthy)
  p50 =   40.0 ms   p95 =   71.1 ms   p99 =   90.3 ms
  p99 inflation from one slow worker: 2.63x

Little's law  L = lambda * W
  throughput lambda =     5000 req/s
  mean latency  W   =     46.8 ms
  concurrency   L   =    233.9 requests in flight

Two lessons fall out of one run. First, the straggler that touches only 2 percent of traffic barely registers in the median or the mean but more than doubles the p99: the tail is exquisitely sensitive to a small fraction of slow work, which is exactly why a single misbehaving worker among hundreds can degrade a whole fleet. Second, Little's law converts a business requirement (5000 requests per second) and a measured latency into a hard concurrency floor (about 234 simultaneous requests) that the fleet must be provisioned to hold. If a single replica can hold only, say, 30 concurrent requests in its memory, you need at least eight replicas just to satisfy the law, before any margin for the tail. This is how an abstract metric becomes a fleet-sizing decision, a thread we pick up quantitatively in Chapter 3.

# From your own measurements: every quantile in one call.
import numpy as np
p50, p95, p99 = np.percentile(latencies_ms, [50, 95, 99])

# From a live endpoint: one shell line, p99 reported automatically.
#   hey  -n 20000 -c 200 https://my-model-endpoint/generate
#   wrk  -t8 -c200 -d30s --latency  https://my-model-endpoint/generate
#   vegeta attack -rate=5000 -duration=30s | vegeta report -type='hdrplot'

4. Cost: Dollars Per Unit of Work Beginner

A distributed AI system that meets its throughput and latency targets is still a failure if it costs more than the value it produces. Cost is therefore the third operational metric, and the meaningful form of it is never the raw bill but the cost per unit of useful work: dollars per million training tokens, dollars per thousand served requests, dollars per million generated output tokens. Normalizing by work is what lets you compare a cheap fleet of many small accelerators against an expensive fleet of few large ones, or this month's serving stack against last month's. The headline number for large-model serving is cost per token, and it falls directly out of the price of the hardware and how hard that hardware is working.

The lever that links cost to the previous metrics is utilization: the fraction of the time your expensive accelerators are actually doing useful work rather than idling on a communication barrier, waiting for the next batch, or sitting in a warm pool. If a GPU that costs $D$ dollars per hour produces useful work only a fraction $u$ of the time, your effective cost per unit of work is inflated by $1/u$. Concretely, if the hardware can produce throughput $X_{\max}$ tokens per second when fully busy but runs at utilization $u$, the realized cost per token is

Halving utilization doubles the cost of every token, which is why so much of distributed serving is a fight to keep accelerators busy through batching, autoscaling, and request routing. This formula also exposes a tension with the latency metric: the cheapest possible serving runs every accelerator at 100 percent utilization with enormous batches, but large batches and full queues push latency, especially the tail, up. Cost and tail latency pull in opposite directions, and choosing the operating point between them is one of the central design decisions in Chapter 24. The unit economics of a single node, which set $X_{\max}$ and $D$ in the first place, are the explicit subject of Chapter 22, the one place this book treats single-node efficiency as a topic in its own right, precisely because fleet cost is per-node cost multiplied across the fleet.

5. Reliability: The Metric That Gets Worse as You Scale Intermediate

The fourth metric is the one that distinguishes distributed systems from single machines most sharply, and it moves in the wrong direction as you scale. Reliability is usually expressed as availability, the fraction of time the system is up and serving, and as a failure rate, how often individual components break. On one machine, you can treat hardware as reliable to a first approximation and spend your attention elsewhere. Across many machines you cannot, because failures compound. Suppose each machine is independently up with probability $r$ (say $r = 0.999$, a respectable three-nines per machine). A job that needs all $M$ machines to be simultaneously healthy is up only with probability

which decays toward zero as $M$ grows. At $r = 0.999$, a 10-machine job is healthy 99.0 percent of the time, a 100-machine job 90.5 percent, and a 1000-machine job only 36.8 percent of the time. A synchronous training run on a thousand GPUs, where one dead worker stalls the whole all-reduce, is therefore in a failed state nearly two thirds of every interval if you do nothing about it. This is not a pessimistic edge case; it is arithmetic, and it is why fault tolerance is not an optional add-on but a structural requirement that threads through the entire book.

The response is to break the "all machines must be healthy" assumption. Replication lets a service survive the loss of any single replica, turning a product of probabilities into a far gentler one: a service that needs only $k$ of $M$ replicas alive can push availability up toward as many nines as you are willing to pay for. Checkpointing lets a training job restart from a recent saved state instead of from zero, bounding the cost of any one failure. Elastic membership lets a job continue with fewer workers rather than halting when one dies. Each of these is a chapter-sized topic: MapReduce re-execution in Chapter 6, elastic and fault-tolerant training in Chapter 18, and the reliability and security of the whole fleet, including failures that are adversarial rather than random, in Chapter 35. The lesson for now is that reliability is a metric you must actively engineer upward, against a baseline that the very act of scaling out drives down.

We now have the four yardsticks the rest of the book measures against: throughput (and its honest cousin goodput), latency at the tail, cost per unit of work scaled by utilization, and reliability that decays with machine count until you engineer it back up. With the metrics fixed, the natural next question is what real systems look like when all four pressures act at once. The next section tours concrete distributed AI systems, from web-scale search to multi-agent platforms, and shows how each one sits at a particular point in the space these four metrics define. That tour begins in Section 1.7.