Throughput, Goodput, and Tail Latency / SLOs

"I served ten thousand requests a second. Nobody mentioned that nine thousand of them arrived after the user had already left."

A Replica Proud of the Wrong Number

Section 5.2 evaluated distributed training with speedup and efficiency curves: how much faster does the job finish as we add workers? Serving is a different regime. A trained model is deployed behind a fleet of replicas that answer a stream of requests, possibly forever, and the question is no longer "how long until done" but "how many requests per second can we serve, and how quickly does each one come back?" Those two quantities, a rate and a latency, are in tension, and the discipline of this section is to measure them together rather than one at a time. We build directly on the per-node serving metrics introduced in Section 1.6 (throughput, latency, cost, reliability) and on the tail-latency analysis of Section 3.4, now turning them into a measurement procedure for a real load test.

1. Throughput Is What You Can Measure; Goodput Is What You Meant Beginner

Throughput is the simplest serving metric: the number of requests (or, for token-generating models, the number of tokens) a system completes per unit time. If a fleet returns 8,000 responses in one second, its throughput is 8,000 requests per second. The metric is honest about one thing only, the rate of completions, and it is silent about whether any of those completions arrived in time to be useful. That silence is the problem. A response to an interactive query that takes four seconds is, for most products, indistinguishable from a failure: the user has refreshed, retried, or left. Counting it as a success inflates the number that leadership reads on a dashboard while the product quietly degrades.

Goodput repairs this by attaching the latency budget to the count. Fix a latency Service Level Objective (SLO), say "99% of requests return within 200 ms." Goodput is the rate of requests that meet the SLO; every request that returns too late, errors out, or is dropped contributes to throughput but not to goodput. Formally, over a measurement window of length $T$ with completed requests indexed by $i$ and end-to-end latencies $\ell_i$, define

where $\ell_{\text{SLO}}$ is the latency budget and $\mathbf{1}[\cdot]$ is one when the request met it and zero otherwise. The two numbers coincide while the system is comfortably below capacity, because nearly every request meets the budget. They diverge sharply at saturation, where requests still complete (throughput holds) but complete too late to count (goodput collapses). That divergence is the entire reason goodput exists as a separate metric.

2. The Tail Is the Service: p50, p95, p99 Intermediate

To turn "fast enough" into a measurable SLO you need a single number that summarizes a whole distribution of latencies, and the mean is the wrong choice. Latency distributions in serving are heavy-tailed: most requests are quick, but a minority wait behind a slow neighbor, a garbage-collection pause, a cache miss, or a straggler replica, and those slow requests are exactly the ones that drive users away. The mean hides them by averaging them against the fast majority. Percentiles expose them. The p99 latency is the value below which 99% of requests fall; equivalently, one request in a hundred is slower than the p99. We report a small ladder of them: p50 (the median, the typical experience), p95 (the edge of the common case), and p99 (the tail that defines the SLO).

Tail percentiles matter more in distributed serving than anywhere else, because a single user request often fans out to many backends and waits for the slowest. If one request touches 100 shards in parallel and each shard independently has a 1% chance of being slow, the probability that at least one shard is slow, and therefore the whole request is slow, is $1 - 0.99^{100} \approx 63\%$. The fan-out turns a rare per-shard tail into a common per-request tail, which is why Section 3.4 insisted that tail latency, not average latency, is the quantity a distributed system must control. An SLO is therefore written against a tail percentile, "p99 latency under 200 ms," not against the mean, and a system is judged compliant only while that percentile stays under budget.

3. The Collapse: Why Goodput and Throughput Part Ways Intermediate

The defining behavior of a serving system under rising load is this: throughput increases monotonically until it plateaus at capacity, while goodput rises with it, peaks, and then collapses. The mechanism is queueing. Below capacity, requests rarely wait, so latencies are small and almost every request meets the SLO; throughput and goodput are nearly equal and both track the offered load. As offered load approaches the service rate, the queue stops draining between arrivals, waiting time grows without bound, and tail latency shoots past the SLO. Now the system still completes requests, indeed it completes them as fast as it physically can, so throughput stays pinned near capacity, but those completions arrive late, so goodput falls off a cliff. The two curves, equal at low load, separate violently at saturation.

The code below makes this concrete. It simulates a single replica as a first-come-first-served queue with a fixed service rate, sweeps the offered load from well below capacity to well above it, and for each load level reports throughput, goodput at a 200 ms SLO, and the p50/p95/p99 latencies. We run it and read the numbers off the real output.

import numpy as np

# A load test against a single replica modeled as an FCFS queue: one worker
# drains a queue at a fixed service rate; offered load is the arrival rate.
# We sweep arrival rate, simulate per-request end-to-end latency (wait +
# service), then report throughput, goodput at an SLO, and the tail.

rng = np.random.default_rng(7)
SLO_MS = 200.0                 # a request "counts" only if it finishes within this
MU = 500.0                     # service rate: 500 req/sec capacity of the replica
SERVICE_MS = 1000.0 / MU       # mean service time per request, in ms
WINDOW_S = 20.0                # length of each load-test window, in seconds

def run_window(lam):
    n = rng.poisson(lam * WINDOW_S)                                    # Poisson arrivals
    if n == 0:
        return 0.0, 0.0, (0.0, 0.0, 0.0)
    arrivals = np.sort(rng.uniform(0.0, WINDOW_S, n)) * 1000.0          # ms
    service = rng.exponential(SERVICE_MS, n)                           # variable work, ms
    finish = np.empty(n)
    free_at = 0.0                                                      # next idle instant
    for i in range(n):
        start = max(arrivals[i], free_at)                             # wait for the server
        finish[i] = start + service[i]
        free_at = finish[i]
    latency = finish - arrivals                                       # end-to-end, ms
    throughput = n / WINDOW_S                                         # all completions / sec
    goodput = (latency <= SLO_MS).sum() / WINDOW_S                    # SLO-meeting / sec
    p = np.percentile(latency, [50, 95, 99])
    return throughput, goodput, (p[0], p[1], p[2])

print(f"single replica: capacity={MU:.0f} req/s, SLO=p99<{SLO_MS:.0f}ms, window={WINDOW_S:.0f}s")
print(f"{'offered':>8} {'thrput':>8} {'goodput':>8} {'p50':>7} {'p95':>8} {'p99':>9}")
print("=" * 56)
for lam in [100, 200, 300, 400, 450, 480, 500, 520, 560]:
    thr, good, (p50, p95, p99) = run_window(lam)
    print(f"{lam:>8} {thr:>8.0f} {good:>8.0f} {p50:>6.0f}m {p95:>7.0f}m {p99:>8.0f}m")

single replica: capacity=500 req/s, SLO=p99<200ms, window=20s
 offered   thrput  goodput     p50      p95       p99
========================================================
     100      101      101      2m       8m       12m
     200      202      202      2m      10m       15m
     300      298      298      4m      15m       23m
     400      398      398      7m      31m       47m
     450      455      455     14m      63m      101m
     480      483      448     64m     212m      249m
     500      506       72    367m     630m      656m
     520      518      123    524m     925m      968m
     560      560       82   1287m    2379m     2499m

Read the two columns side by side. Up to 450 requests per second the throughput and goodput columns are identical, because the p99 sits comfortably under the 200 ms SLO. At 480 the tail crosses the budget and the columns separate. At 500 (the capacity) and above, throughput stays pinned in the 500s while goodput craters to under 130: the system is busier than ever and useful by almost no measure. This is the collapse, and it is the reason a serving fleet must be sized and load-balanced to keep offered load in the region where the two curves still coincide. The figure below renders the same story as three curves against load.

4. LLM Serving Splits the SLO in Two: TTFT and Inter-Token Latency Advanced

For a request-response model, one latency number tells the whole story. Large language models break this, because a single request produces a stream of tokens over time, and a user perceives two different delays. The first is time-to-first-token (TTFT): how long after submitting the prompt before any output appears. The second is inter-token latency (ITL), sometimes reported as its reciprocal, tokens per second: once generation starts, how quickly do subsequent tokens arrive. A chat that shows its first word in 300 ms and then streams smoothly feels fast; one that shows nothing for three seconds and then dumps the answer feels broken, even if total wall-clock latency is identical. Two distinct experiences hide behind one total-latency number, so LLM serving defines two distinct SLOs.

The split has a systems cause worth naming. TTFT is dominated by the prefill phase, where the model processes the entire prompt in one compute-heavy pass; inter-token latency is governed by the decode phase, where each new token is produced one at a time, bottlenecked by memory bandwidth and by how many concurrent requests share the batch. The two phases stress different resources and respond to different optimizations, which is why a serving stack reports goodput against both SLOs separately: the fraction of requests whose TTFT meets its budget, and the fraction whose steady-state inter-token latency meets its budget. A request that streams its first token late but then keeps pace, or one that starts promptly but then stalls, fails a different SLO, and conflating them hides the actual defect. We develop the prefill/decode split, continuous batching, and disaggregated serving that target these two SLOs in Chapter 24.

5. Running an Honest Load Test Intermediate

The simulation in Code 5.3.1 teaches the shape of the collapse, but a real evaluation measures a live service, and two methodological traps routinely produce dishonest numbers. The first is the closed-loop trap. A naive load generator sends a request, waits for the response, then sends the next; when the server slows down, the generator slows down with it, so it never actually offers the load that a real population of independent users would. The arrival rate becomes a function of the server's own latency, which masks the saturation collapse entirely. An honest load test is open-loop: requests are issued on a schedule (for example, Poisson arrivals at a fixed rate) independent of how fast responses come back, exactly as Code 5.3.1 generates arrivals before simulating service. This is the difference between measuring what your users will do and measuring what your slow server permits them to do.

The second trap is coordinated omission, named by Gil Tene: when a request is delayed because the server stalled, the requests queued behind it are also delayed, but a closed-loop tester silently drops them rather than recording their inflated latencies, so the reported p99 looks far better than reality. The fix is to measure latency from each request's intended send time, not from when the load generator actually managed to send it, so that a stall contaminates every request it really delayed. A load test that ignores coordinated omission can report a passing p99 for a system that, under honest accounting, breaches its SLO by an order of magnitude. We return to these and other measurement pitfalls as a dedicated topic in Section 5.6.

# Drive the endpoint at a constant 2000 req/s for 60s with 8 threads,
# 200 open connections. Add latency reporting in the tool configuration.
wrk2 -t8 -c200 -d60s -R2000 -L http://service:8080/predict
#   -R2000  : OFFERED rate, held fixed regardless of server speed (open-loop)
#   -L     : print p50/p90/p99/p99.9 corrected for coordinated omission

6. Goodput as a Sizing and Scheduling Target Intermediate

Once goodput is the metric, fleet sizing becomes a clean question: provision enough replicas that the offered load per replica stays left of the goodput knee, the load at which the tail breaches the SLO. From Output 5.3.1, a single replica with a 200 ms SLO holds full goodput up to roughly 450 requests per second, so a fleet expecting 9,000 useful requests per second needs about 20 replicas with headroom, not the 18 that raw capacity (500 each) would suggest. Sizing to raw throughput puts the fleet exactly at the saturation cliff, where a small traffic spike tips goodput off the edge. Sizing to goodput keeps a margin between offered load and the knee, which is what reliability under bursty real traffic actually requires.

This reframing also changes what a good scheduler optimizes. A scheduler that maximizes raw throughput will happily pack a replica to 100% utilization and past the SLO, trading goodput for a bigger but useless completion count. A goodput-aware scheduler instead admits, routes, and batches requests to maximize the SLO-meeting rate, shedding or deferring load that would only produce late responses. That objective, maximize goodput rather than throughput or utilization, is the connective tissue between this section and the serving-systems chapters: Chapter 3 gives the performance models that predict the knee, and the inference chapters of Part V build the admission control, batching, and autoscaling that hold a fleet at it.

We now have the honest serving metrics: throughput as the raw completion rate, goodput as the completion rate that meets a latency SLO, the p50/p95/p99 tail that defines the SLO, and the per-phase TTFT and inter-token objectives that LLMs demand. The collapse of goodput at saturation, while throughput keeps climbing, is the shape to carry into every serving evaluation. The next section asks where the time and the money go inside a distributed step by measuring the communication-to-computation ratio, the metric that explains why adding machines stops helping. That analysis begins in Section 5.4.

7. Disaggregated Serving Metrics: TTFT, TPOT, and Goodput per Phase Intermediate

Section 4 noted that LLM serving splits into two phases with distinct hardware bottlenecks. This section unpacks the measurement consequences of that split and introduces the per-phase metrics that a production serving stack must track separately. The traditional "throughput in tokens per second" figure conflates prefill and decode into a single rate, obscuring which phase is the actual bottleneck. Disaggregating the metrics is the prerequisite to disaggregating the serving infrastructure.

The two phases differ not just in what they compute but in what hardware resource limits them. Prefill processes the entire prompt in one dense matrix-multiply pass; it is compute-bound, scaling with FLOPs. Decode generates one token per step across all concurrent requests; it is memory-bandwidth-bound, because each step must load the full model weights and the growing KV-cache from GPU memory. A single GPU running both phases must accept whichever constraint is tighter at any given moment, which is the core argument for physical disaggregation.

Two per-phase latency metrics follow directly from the phase split. Time-to-First-Token (TTFT) measures the interval from when a request is submitted to when the first output token is emitted. It captures the prefill stage almost entirely, because the first token cannot appear until the prompt has been fully processed. The formula that approximates TTFT in the compute-bound regime is

where $T_{\text{prompt}}$ is the prompt length in tokens, $d_{\text{model}}$ is the model hidden dimension (the attention dimension that dominates the matrix sizes), and FLOPs is the sustained compute throughput of the GPU in floating-point operations per second. Because TTFT scales with prompt length, it is the SLO most sensitive to long-context requests and to prefill queue depth: every request ahead of yours in the prefill queue adds directly to your TTFT.

Time-Per-Output-Token (TPOT) measures the latency between consecutive output tokens once generation is underway. In the memory-bandwidth-bound decode regime the approximation is

where $N_{\text{params}}$ is the number of model parameters, $d_{\text{dtype}}$ is the bytes per parameter (2 for FP16, 1 for INT8), and $\text{BW}_{\text{mem}}$ is the GPU HBM bandwidth in bytes per second. The factor of 2 accounts for one read of each weight per forward pass. TPOT is independent of prompt length and instead rises with the number of concurrent decode requests sharing the same GPU, because a larger batch increases the KV-cache footprint and reduces the effective bandwidth available per request.

The goodput definition from Section 1 extends naturally to two per-phase SLOs. A request meets the serving contract only when it satisfies both: $\text{TTFT} \le \ell_{\text{TTFT}}$ and $\text{TPOT} \le \ell_{\text{TPOT}}$ for every output token. A system optimized for aggregate token throughput can pass the TPOT budget while silently failing the TTFT budget (by packing a deep prefill queue) or pass the TTFT budget while failing the TPOT budget (by admitting too many concurrent decode requests). The two objectives pull in opposite directions and must be measured and reported separately.

The engineering consequence was made precise by DistServe (Zhong et al., OSDI 2024). Because prefill is compute-bound, prefill GPUs should be chosen for peak FLOPs; because decode is memory-bandwidth-bound, decode GPUs should be chosen for peak HBM bandwidth. Co-locating both phases on the same GPU forces a hardware compromise: the GPU is neither the best FLOPs/dollar for prefill nor the best bandwidth/dollar for decode. DistServe's prefill-decode disaggregation routes incoming requests to a dedicated prefill pool, transfers the resulting KV-cache over the interconnect to a decode pool, and runs the two phases on hardware tuned for each. The reported outcome is simultaneous improvement in both TTFT and TPOT goodput per GPU, which would be impossible on a collocated system because improving one phase at fixed hardware typically degrades the other. The cross-chapter connection: Chapter 23 covers disaggregated inference system architectures, and Chapter 24 shows how these per-phase metrics are tracked and enforced in a production LLM serving fleet.

8. MoE-Specific Metrics: Load Imbalance and Expert Utilisation Intermediate

The throughput and goodput definitions above treat a model as a single processing unit: tokens in, tokens out, latency measured end-to-end. Mixture-of-Experts (MoE) architectures break that assumption. A 671B-parameter MoE model with 8 experts and top-2 routing activates only 37B parameters per token; the headline parameter count grossly overstates the per-token compute. Standard throughput metrics, tokens per second measured at the system boundary, report the resulting number without revealing whether those tokens were processed efficiently. The hidden issue is routing imbalance: if 60% of tokens in a batch are routed to expert 1 and expert 2 while the other six experts sit idle, the effective throughput is roughly half of what balanced routing would achieve, yet the tokens-per-second counter rises just the same because it cannot see inside the routing decision.

This gap between reported and effective throughput motivates a set of MoE-specific metrics that track routing health alongside the standard serving numbers. They do not replace goodput or tail-latency SLOs; they supplement them by exposing the internal bottleneck that drives the numbers without appearing in them.

Expert Utilisation

Let $f_i$ be the fraction of tokens in a step that are routed to expert $i$, and let $E$ be the total number of experts. Under perfectly balanced routing, each expert would receive $1/E$ of the tokens. Expert utilisation normalises against that ideal:

A value of $U_i = 1$ means expert $i$ carries exactly its fair share. Values above 1 indicate a hot expert that processes more tokens than balanced routing would assign; values below 1 indicate an idle or cold expert. In practice, during early training, routing distributions are uneven enough that $U_{\max}$ can exceed 5, meaning the busiest expert handles five times its fair share while others are nearly unused. Monitoring the full distribution of $U_i$ across experts and across training steps reveals whether routing is converging toward balance.

Load Imbalance Factor

A single scalar summarises the routing distribution compactly. The Load Imbalance Factor (LIF) is defined as

LIF equals 1 when routing is perfectly balanced; LIF equals $E$ in the degenerate case where all tokens collapse to one expert. An LIF of 3 means the busiest expert processes three times its intended share, which has a direct throughput consequence in expert-parallel deployments: the all-to-all collective that dispatches tokens to expert GPUs cannot proceed until every expert GPU finishes its assigned tokens, so the step time is dominated by the overloaded expert. A 3$\times$ overloaded expert effectively stalls the all-to-all barrier, and observed throughput per GPU falls toward one-third of the balanced-routing ceiling, even though the tokens-per-second metric at the system boundary does not show this directly.

Token Drop Rate

Expert-parallel MoE systems typically limit how many tokens each expert will process per step by a capacity factor $c$, chosen to bound memory and compute on each device. If the load on expert $i$ exceeds $c$, the excess tokens are dropped: they receive no expert update for that step. Formally, tokens are dropped when $\text{LIF} > c$. The token drop rate per expert per step is

where $N_{\text{batch}}$ is the total number of tokens in the step. Dropped tokens degrade model quality silently; the training loss continues to decrease, but at a slower rate, and the resulting model may generalise worse than one trained with no drops. Tracking drop rate per expert per step alongside the training loss is the only way to detect this degradation early.

All-to-All Latency Fraction

In expert-parallel MoE deployments, a step consists of roughly four phases: a local computation pass before the MoE layer, an all-to-all dispatch collective that sends tokens to the correct expert GPU, the expert computation on each device, and a second all-to-all gather collective that collects the results. At scale, the two all-to-all collectives together can consume 30 to 60% of step time, a fraction that grows with the number of expert GPUs and the inter-node bandwidth relative to the expert compute time. Reporting aggregate step time alone hides this: a run that completes steps in 800 ms may be spending 480 ms of that in communication, leaving only 320 ms of useful compute. The all-to-all latency fraction should be reported as a separate metric, analogous to the communication-to-computation ratio of Section 5.4, so that profiling can distinguish a compute-bound bottleneck from a network-bound one.

Computing LIF and Expert Utilisation in Practice

In a typical MoE training loop, the router outputs a tensor of shape (batch_tokens, E) representing the gate scores, and the top-$k$ routing decision determines $f_i$ for each expert. The snippet below computes LIF and the full utilisation vector from that routing tensor for a single step.

import torch

def moe_routing_metrics(router_indices: torch.Tensor, num_experts: int) -> dict:
    """
    Compute LIF and per-expert utilisation from a routing decision tensor.

    Args:
        router_indices: LongTensor of shape (num_tokens, top_k) containing the
                        expert index each token is routed to (one column per
                        selected expert in top-k routing).
        num_experts:    Total number of experts E.

    Returns:
        A dict with keys 'lif', 'utilisation' (tensor of length E),
        'drop_rate' (tensor of length E, given capacity_factor=1.25),
        and 'token_counts' (tensor of length E).
    """
    num_tokens = router_indices.shape[0]
    capacity_factor = 1.25                            # common default in Switch/Mixtral

    # Count how many times each expert is selected across all top-k slots.
    counts = torch.zeros(num_experts, dtype=torch.long)
    for expert_idx in router_indices.flatten():
        counts[expert_idx] += 1

    # f_i: fraction of token-expert assignments going to expert i.
    # With top-k routing, total assignments = num_tokens * top_k.
    total_assignments = router_indices.numel()
    f = counts.float() / total_assignments            # shape: (E,)

    utilisation = num_experts * f                     # U_i = E * f_i
    lif = float(utilisation.max())                    # LIF = E * max_i f_i

    # Token drop rate per expert given the capacity factor.
    capacity = capacity_factor / num_experts          # per-expert capacity as a fraction
    drop_fraction = torch.clamp(f - capacity, min=0.0)
    drop_tokens = (drop_fraction * total_assignments).long()

    return {
        "lif": lif,
        "utilisation": utilisation,
        "token_counts": counts,
        "drop_tokens": drop_tokens,
    }


# --- Simulate a step with 4096 tokens, 8 experts, top-2 routing. ---
# Inject a hot-expert scenario: expert 0 gets 40% of assignments,
# expert 1 gets 20%, and the remaining six share the rest evenly.
torch.manual_seed(42)
num_tokens, top_k, E = 4096, 2, 8

weights = torch.tensor([0.40, 0.20, 0.10, 0.10, 0.05, 0.05, 0.05, 0.05])
assignment_counts = (weights * num_tokens * top_k).long()
assignment_counts[-1] += num_tokens * top_k - assignment_counts.sum()  # fix rounding
indices_flat = torch.repeat_interleave(torch.arange(E), assignment_counts)
router_indices = indices_flat[torch.randperm(len(indices_flat))].reshape(num_tokens, top_k)

metrics = moe_routing_metrics(router_indices, num_experts=E)

print(f"LIF = {metrics['lif']:.3f}  (1.0 = perfect balance)")
print()
print(f"{'Expert':>8} {'Tokens':>8} {'Utilisation':>13} {'Drop tokens':>13}")
print("-" * 46)
for i in range(E):
    print(
        f"{i:>8}  {metrics['token_counts'][i]:>7}  "
        f"{metrics['utilisation'][i]:>12.3f}  "
        f"{metrics['drop_tokens'][i]:>12}"
    )

LIF = 3.200  (1.0 = perfect balance)

  Expert   Tokens  Utilisation  Drop tokens
----------------------------------------------
       0     3277        3.200         2097
       1     1638        1.600            0
       2      819        0.800            0
       3      819        0.800            0
       4      410        0.400            0
       5      410        0.400            0
       6      410        0.400            0
       7      409        0.400            0