Why Large-Model Serving Spans Many Machines

"I was asked to hold the model and remember the conversation. I could do one of those. They sent for more of me."

A GPU That Ran Out of Memory Mid-Sentence

The single-node mechanisms that make a transformer cheap to run, the key-value cache that turns quadratic decoding into linear decoding, paged attention that stops that cache from fragmenting memory, continuous batching that keeps the accelerator busy, and speculative decoding that spends a small model to save the big one, were the subject of Chapter 22. We treated them there as the per-node prerequisite, the scale-up baseline that distribution multiplies. This chapter assumes them and asks the next question. Once you have wrung out one GPU, what do you do when the model and its working memory still will not fit, or when one GPU cannot meet the latency a user expects? The answer is to span machines, and the shape of that span follows directly from where the memory goes.

1. One Request Already Overflows One GPU Beginner

Start with a single user sending a single prompt. Two pools of memory must coexist on the accelerator. The first is the model weights, fixed in size: a model with $P$ parameters stored at $b$ bytes each occupies $P b$ bytes regardless of how many requests are in flight. The second is the key-value cache, the per-request memory that holds the attention keys and values for every token already seen, so that each new token attends to the past without recomputing it. The cache grows with the conversation. For a model with $L$ layers and hidden width $H$, every token stores keys and values across all layers, costing

where the leading $2$ counts keys and values separately and $b_{\text{kv}}$ is the bytes per stored number. The total memory a single GPU of capacity $M$ must hold for a batch of $R$ concurrent requests each of context length $C$ is therefore

The weights term is a one-time cost; the cache term scales with both how long the conversations are and how many of them run at once. The crucial observation, the one that forces this entire chapter, is that for a large model the weights alone can exceed $M$, and even when they do not, the cache term overtakes them as context lengths grow into the tens or hundreds of thousands of tokens. Neither term is optional: you cannot drop the weights, and you cannot answer a long prompt without remembering it.

The fix for the weights term is to shard the model across GPUs so that each device holds only a slice of every layer and the devices cooperate on each forward pass. Splitting a single matrix multiply across GPUs and summing their partial results is tensor parallelism, the subject of Section 24.2, and the summation it relies on is the same all-reduce collective introduced for training in Chapter 4. The fix for the cache term, when the cache outgrows the GPUs that hold the weights, is to distribute the key-value cache itself, which Section 24.4 develops. Before either fix, it pays to see the overflow in numbers.

2. The Fit Calculator: Counting GPUs From Memory Intermediate

The inequality above is small enough to evaluate directly, and doing so turns "the model is too big" from a slogan into a count of machines. Given the weight bytes and the per-token cache cost, the minimum number of GPUs is the total memory divided by one GPU's capacity, rounded up,

This is a lower bound: it assumes weights and cache can be packed perfectly across devices with no replication or overhead, which real tensor-parallel layouts only approximate. Even as a lower bound it is decisive. The code below evaluates it for three model sizes at two operating points, a short prompt with modest concurrency and a long-context workload with high concurrency, using bf16 weights and an 80 GB accelerator.

import math

# (name, params_billions, hidden H, layers L)
models = [("8B", 8, 4096, 32), ("70B", 70, 8192, 80), ("405B", 405, 16384, 126)]
B_WEIGHT, B_KV, GPU_GB = 2, 2, 80          # bf16 weights, fp16 KV, one 80 GB GPU

def kv_bytes_per_token(H, L):              # keys + values, all layers
    return 2 * L * H * B_KV

# (context length C, concurrent requests R)
points = [(8_000, 32), (128_000, 256)]

print(f"{'Model':>6} {'ctx':>8} {'reqs':>5} {'weights GB':>11} {'KV GB':>9} {'total GB':>9} {'GPUs':>6}")
for name, pb, H, L in models:
    w = pb * 1e9 * B_WEIGHT
    for C, R in points:
        kv = kv_bytes_per_token(H, L) * C * R
        total = w + kv
        gpus = math.ceil(total / (GPU_GB * 1e9))     # the memory lower bound on devices
        print(f"{name:>6} {C:>8} {R:>5} {w/1e9:>11.1f} {kv/1e9:>9.1f} {total/1e9:>9.1f} {gpus:>6}")

 Model      ctx  reqs  weights GB     KV GB  total GB   GPUs
    8B     8000    32        16.0     134.2     150.2      2
    8B   128000   256        16.0   17179.9   17195.9    215
   70B     8000    32       140.0     671.1     811.1     11
   70B   128000   256       140.0   85899.3   86039.3   1076
  405B     8000    32       810.0    2113.9    2923.9     37
  405B   128000   256       810.0  270582.9  271392.9   3393

Read the first column of numbers and the thesis of the chapter falls out. The 8B model has only 16 GB of weights, comfortably inside one 80 GB GPU, yet at a 32-request batch of 8,000-token contexts it already needs two GPUs because the cache adds 134 GB. The 70B model's weights alone, at 140 GB, overflow a single device before any user connects. And the cache term is explosive: at 128,000-token contexts with 256 concurrent requests, the 70B model's cache reaches roughly 86 terabytes, swamping its weights by a factor of six hundred and demanding over a thousand GPUs. The lesson the table teaches, and the reason large-model serving is a distributed-systems problem rather than a single-box optimization, is that the memory you must reserve to remember conversations grows without bound while the weights stay fixed. Section 24.4 returns to this table when it distributes the cache itself.

3. Sharding the Model, and the Two Phases of a Request Intermediate

Because one GPU cannot hold the model, the forward pass is spread across several. Each device holds a slice of every weight matrix, performs its part of each layer's matrix multiplications, and the devices exchange partial results so that the layer's output is correct as if one machine had computed it. Figure 24.1.1 shows the progression: a model and its growing cache overflowing one GPU on the left, the same model sharded across four cooperating GPUs in the middle, and on the right the two phases of generation that, at fleet scale, motivate splitting the work even further across machines.

The right panel of Figure 24.1.1 introduces the structural fact that the rest of the chapter exploits. Generating an answer is not one uniform computation; it is two phases. In the prefill phase the model reads the entire prompt at once, computing keys and values for every prompt token in parallel and producing the first output token. Prefill is compute-bound: it performs large matrix multiplications over many tokens simultaneously and saturates the accelerator's arithmetic units. In the decode phase the model generates the rest of the answer one token at a time, and each step must read the entire key-value cache to attend to the past while doing comparatively little arithmetic. Decode is memory-bound: it is limited by how fast the device can stream the cache out of memory, not by how fast it can multiply. One request, two opposite bottlenecks.

4. Two SLOs, Not One: TTFT and TPOT Intermediate

Because a request has two phases, its quality of service has two numbers, and collapsing them into a single latency figure hides the trade-off that distributed LLM serving exists to manage. The first is time-to-first-token (TTFT), the delay from when the user submits a prompt to when the first output token appears. TTFT is set almost entirely by prefill: a long prompt or a busy prefill queue makes the user wait before anything streams. The second is time-per-output-token (TPOT), sometimes called inter-token latency, the average gap between successive tokens once generation has begun. TPOT is set by decode: it determines how fast the answer streams out, and a user reads comfortably when it stays below roughly the speed of reading aloud.

These two objectives pull in different directions. Batching more requests together raises decode throughput, which is good for serving many users cheaply, but a larger batch lengthens each decode step and inflates TPOT for everyone in it. Prioritizing a new request's prefill to lower its TTFT steals accelerator time from the decode steps of requests already streaming, raising their TPOT. A serving system that optimizes one number in isolation will quietly wreck the other. This is why the right success metric is not latency but goodput, the rate of requests served while meeting both SLOs at once, a notion Section 5.3 defined for distributed systems in general and that Chapter 24 specializes to the TTFT-and-TPOT pair.

5. The Road Through This Chapter Beginner

The memory inequality and the two SLOs together set the agenda for everything that follows, and it is worth seeing the whole route before taking the first step. Section 24.2 shards a single forward pass across GPUs with tensor parallelism, the direct fix for weights that overflow one device. Section 24.3 extends the split across nodes with pipeline parallelism when even a tensor-parallel group spans more machines than one server holds. Section 24.4 distributes the key-value cache itself, the term that dominated Output 24.1.1. Section 24.5 disaggregates prefill from decode, the architectural answer to the two-SLO tension. Section 24.6 schedules requests and runs continuous batching across nodes so the fleet stays busy without violating either SLO. Section 24.7 reuses shared prompt prefixes and serves many fine-tuned adapters from one fleet. Section 24.8 serves mixture-of-experts models whose experts live on different machines, the inference companion to the expert parallelism of Chapter 17. Section 24.9 grounds all of it in the production engines that implement these ideas.

from vllm import LLM, SamplingParams

# tensor_parallel_size shards every layer across 4 GPUs; the engine wires up the
# per-layer all-reduce, paged KV cache, and continuous batching for you.
llm = LLM(model="meta-llama/Llama-3.1-70B-Instruct", tensor_parallel_size=4)

out = llm.generate(["Summarize this support transcript: ..."],
                   SamplingParams(max_tokens=256))
print(out[0].outputs[0].text)

We have the reason a single machine is impossible, the calculator that turns it into a GPU count, and the two service-level objectives that every later design must honor. The most direct response to weights that do not fit is to split one matrix multiply across devices and combine their answers, and that is exactly where the next section begins, with tensor-parallel inference in Section 24.2.