Distributed Model Serving with vLLM

"A KV cache, paged like virtual memory, serving a hundred conversations at once and pretending each one is the only one that matters."

A Decode Step That Never Stops to Wait

The agents of Section 40.6 are clients. They plan, call tools, and reflect, but the substance of every step is a call to a large language model that returns the next action as text. In a toy demo that model is a hosted API behind a key. In the system this chapter builds, the application owns its serving tier, because the agent fleet issues far too many calls, with too much shared structure and too strict a cost target, to leave on a metered public endpoint. Owning the backend means owning the engineering that Chapter 24 teaches: continuous batching, paged key-value (KV) caches, prefill and decode scheduling, and tensor parallelism, all wrapped in a replicated, autoscaling fleet. vLLM is the open engine that packages those mechanisms, and it is the backend behind a large share of self-hosted agent deployments today.

The shape of the agentic workload makes the choice of engine consequential rather than incidental. A web-serving stack treats requests as independent and roughly uniform. Agent traffic is neither: a single user task spawns a tree of model calls (plan, then several tool-formatting calls, then a reflection), most of them short, a few of them long, and almost all of them prefixed by an identical multi-thousand-token system prompt that names the tools and the rules. A serving engine that exploits that structure, by batching across requests continuously and by storing the shared prefix once, serves several times more agents per GPU than one that does not. Figure 40.7.1 shows the fleet we are about to build.

1. The Serving Engine: Continuous Batching and PagedAttention Intermediate

A language model generates one token at a time. Each step runs the full network over the current context and emits the next token, so a request of $T$ output tokens needs $T$ sequential forward passes through the model. Per-node efficiency, the subject of Chapter 22, is about making each of those passes cheap; serving throughput is about running as many requests through each pass as possible. The two combine multiplicatively, which is why the fleet inherits Chapter 22 as a labeled prerequisite rather than re-deriving it.

The naive way to batch is static: collect a fixed group of requests, run them together until the last one finishes, then start the next group. Agent traffic punishes this badly, because the generation lengths vary by an order of magnitude. A batch holding fifteen twenty-token tool calls and one five-hundred-token plan runs for five hundred steps, and for most of those steps the short sequences have already finished and their slots sit idle. Continuous batching, also called in-flight batching, fixes this by scheduling at the granularity of a single decode step: the moment a sequence emits its end-of-sequence token and frees a slot, a waiting request is admitted into that slot on the very next step. The engine never stalls on the slowest member of a batch. If we write $g_i$ for the output length of request $i$ in a window and $d$ for the per-step decode time of the batch, static batching keeps a batch of size $B$ busy for $\max_i g_i$ steps while only $\sum_i g_i$ are useful, so its slot utilization is

and the achieved throughput in tokens per second rises in direct proportion to that utilization. Under the heavy length variance of agent traffic, $U_{\text{static}}$ can fall well below one half, and continuous batching recovers the lost fraction. The demo in Code 40.7.1 measures exactly this gap.

The second mechanism is memory management for the KV cache. During generation, each layer caches the key and value vectors of every past token so they are not recomputed; this cache grows with the context length and dominates the memory budget once the weights are resident. The bytes a single sequence consumes are

where $L$ is the number of layers, $H_{\text{kv}}$ the number of key-value heads, $d_{\text{head}}$ the head dimension, $T$ the context length, $b$ the bytes per element (the factor $2$ counts keys and values). Reserving the maximum possible $T$ for every sequence up front, as early servers did, wastes most of the cache on contexts that never grow that long. PagedAttention, the mechanism vLLM is built around (Kwon et al., 2023), borrows the idea of paged virtual memory from operating systems: it splits the KV cache into fixed-size blocks and allocates them on demand as a sequence grows, so physical memory holds only the tokens that actually exist. Fragmentation drops from roughly sixty percent in the reserve-the-maximum scheme to a few percent, which is the same as multiplying the number of sequences that fit by a factor of two or more. Quantizing the cache to a smaller $b$, the scale-up lever of Chapter 22, shrinks $M_{\text{KV}}$ further.

2. Sizing the Cache: How Many Agents Fit on One GPU? Intermediate

The formulas above turn into a capacity number the moment we plug in a real model and a real card. The demo below does this for a 13-billion-parameter decoder served in half precision on one 80 GB accelerator. It computes the KV bytes per token, subtracts the resident weights and a working-memory reserve to find the budget left for the cache, and divides to get the number of concurrent sequences. It then models the prefix-sharing win that is specific to agents, and finally compares continuous against static batching on a length distribution drawn from agentic traffic. The numbers it prints are the ones you would use to decide how many replicas the fleet needs.

import math, random

# ---- Model / hardware configuration (a 13B-class decoder served in fp16) ----
n_layers, n_kv_heads, head_dim, bytes_per = 40, 40, 128, 2   # fp16 -> 2 bytes
gpu_mem_gb, weights_gb, overhead_gb = 80, 26, 6              # one A100-80GB

# KV-cache bytes per token = 2 (K and V) * layers * kv_heads * head_dim * bytes
kv_per_token = 2 * n_layers * n_kv_heads * head_dim * bytes_per
kv_budget_bytes = (gpu_mem_gb - weights_gb - overhead_gb) * (1024 ** 3)

avg_ctx = 2048                                  # tokens per sequence
kv_per_seq = kv_per_token * avg_ctx
max_seqs = int(kv_budget_bytes // kv_per_seq)
print("=== KV-cache sizing (13B fp16, A100-80GB) ===")
print(f"KV bytes per token        : {kv_per_token:,} B  ({kv_per_token/1024:.1f} KB)")
print(f"Memory free for KV cache  : {gpu_mem_gb-weights_gb-overhead_gb} GB")
print(f"KV bytes per sequence     : {kv_per_seq/1e6:.1f} MB  (ctx={avg_ctx})")
print(f"Max concurrent sequences  : {max_seqs}")

# ---- Prefix sharing: agents share a long system prompt, stored ONCE ----
sys_prompt_tokens, unique_tokens = 1500, 548
shared_once = kv_per_token * sys_prompt_tokens
unique_each = kv_per_token * unique_tokens
seqs_with_caching = int((kv_budget_bytes - shared_once) // unique_each)
print("\n=== Prefix caching for an agent fleet ===")
print(f"Concurrent seqs, no cache : {max_seqs}")
print(f"Concurrent seqs, prefix-cached : {seqs_with_caching}  "
      f"({seqs_with_caching/max_seqs:.2f}x)")

# ---- Continuous (in-flight) batching vs static batching ----
def static(batch, lens, d_ms):
    steps = max(lens)                            # wait for the slowest sequence
    return sum(lens) / (steps*d_ms/1000), sum(lens) / (batch*steps)
def continuous(batch, lens, d_ms):
    return sum(lens) / ((sum(lens)/batch)*d_ms/1000), 1.0   # refill freed slots

random.seed(7)
batch, d_ms = 16, 12
lens = [random.choice([20, 30, 40, 350, 512]) for _ in range(batch)]
s_tps, s_u = static(batch, lens, d_ms)
c_tps, c_u = continuous(batch, lens, d_ms)
print("\n=== Continuous vs static batching (variable agent outputs) ===")
print(f"Static  batching: {s_tps:6.1f} tok/s   slot utilization {s_u:.2f}")
print(f"Continuous batch: {c_tps:6.1f} tok/s   slot utilization {c_u:.2f}")
print(f"Speedup from continuous batching : {c_tps/s_tps:.2f}x")

=== KV-cache sizing (13B fp16, A100-80GB) ===
KV bytes per token        : 819,200 B  (800.0 KB)
Memory free for KV cache  : 48 GB
KV bytes per sequence     : 1677.7 MB  (ctx=2048)
Max concurrent sequences  : 30

=== Prefix caching for an agent fleet ===
Concurrent seqs, no cache : 30
Concurrent seqs, prefix-cached : 112  (3.73x)

=== Continuous vs static batching (variable agent outputs) ===
Static  batching:  463.2 tok/s   slot utilization 0.35
Continuous batch: 1333.3 tok/s   slot utilization 1.00
Speedup from continuous batching : 2.88x

The three numbers tell a coherent story for the fleet planner. Bare capacity is thirty sequences. Because the agents share a long system prompt, prefix caching nearly quadruples that to over a hundred, which is the single largest lever the agentic structure offers. And whatever the concurrency, continuous batching roughly triples the token rate the GPU actually delivers under the bursty, ragged traffic the agents produce. A replica is therefore not "one GPU equals one agent" but "one shard group equals a hundred-odd agents," and the replica count follows from dividing the fleet's peak concurrency by that figure.

3. Prefill, Decode, and the Agentic Traffic Shape Advanced

A generation request has two phases with opposite resource profiles, and the agentic workload makes the split unusually sharp. Prefill processes the entire input prompt in one parallel pass to populate the KV cache; it is compute-bound and its cost grows with the prompt length. Decode then emits output tokens one at a time, each pass reading the whole cache to produce a single token; it is memory-bandwidth-bound and its cost grows with the output length. Agent calls are prefill-heavy: the prompt is a multi-thousand-token system message plus tool schemas plus history, while the output is often a short tool-call object of a few dozen tokens. A scheduler that lets a giant prefill monopolize the engine stalls every in-flight decode behind it, spiking the latency of the short calls that dominate agent traffic.

Two refinements address this. Chunked prefill splits a long prompt into pieces and interleaves them with ongoing decode steps, so a large prompt no longer freezes the batch. Prefill-decode disaggregation, the more aggressive design that Chapter 24 develops, runs the two phases on separate pools of GPUs: a prefill pool optimized for throughput populates the cache, ships the KV blocks to a decode pool optimized for low-latency token streaming, and each pool is sized to its own bottleneck. For an agent fleet, where every call pays a heavy prefill but most calls decode briefly, disaggregation lets the two pools scale independently rather than forcing one GPU profile to straddle both regimes. Prefix caching compounds the benefit: the shared system prompt is prefilled once and its KV blocks are reused across every agent that carries it, so the expensive phase is largely amortized away.

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-2-13b-chat-hf",
          tensor_parallel_size=2,            # shard the model across 2 GPUs
          enable_prefix_caching=True,        # store the shared prompt once
          gpu_memory_utilization=0.92)       # leave headroom for the KV cache
params = SamplingParams(max_tokens=128, temperature=0.0)

system = "You are an agent. Tools: search(q), calc(expr). Rules: ..."  # shared prefix
prompts = [system + f"\nUser task {i}: ..." for i in range(256)]       # one fan-out
for out in llm.generate(prompts, params):     # continuous batching, automatic
    handle(out.outputs[0].text)

vllm serve meta-llama/Llama-2-13b-chat-hf \
    --tensor-parallel-size 2 --enable-prefix-caching \
    --max-num-seqs 128 --port 8000
# agents call http://localhost:8000/v1/chat/completions exactly as a hosted API

4. A Model Too Big for One GPU: Tensor and Pipeline Parallelism Advanced

The capacity arithmetic of Section 2 assumed the weights fit on one card. Many serving models do not. A 70-billion-parameter model in half precision needs roughly 140 GB for weights alone, well beyond a single 80 GB GPU before any KV cache is allocated. The model must then be split across devices, and this is the model-parallel axis that Chapter 16 develops for training and that serving reuses almost unchanged. Tensor parallelism partitions each layer's weight matrices across $N$ GPUs, so each device holds and computes on a $1/N$ slice; the partial results are combined with an all-reduce within every layer, the same collective introduced in Chapter 4 and used for gradients in Chapter 15. If the per-GPU compute time of a layer is $c$ and the all-reduce of its activations costs $a$, the tensor-parallel time per layer is approximately

so adding GPUs shrinks the compute term but inflates the communication term; tensor parallelism therefore stays inside a single node where the interconnect (NVLink) is fast, and the practical degree is small (two, four, or eight). Pipeline parallelism is the complementary split: it assigns whole contiguous layers to different GPUs and streams micro-batches through the stages, which moves far less data per step and so tolerates slower cross-node links, at the cost of pipeline fill-and-drain bubbles. A model too large even for one node combines them, tensor-parallel within each node and pipeline-parallel across nodes, the two-axis layout of Chapter 16. In Figure 40.7.1 each replica is a two-way tensor-parallel group; scaling the model to 70B would deepen each replica to eight-way tensor parallelism, or add a pipeline axis, without changing the fleet structure around it.

5. The Serving Fleet: Replicas, Router, and Autoscaling Intermediate

One replica, however well tuned, has a ceiling: the hundred-odd concurrent agents of Output 40.7.1, at some token rate. The agent fleet at peak wants more, so the replica is copied. Each replica is an identical shard group, and a router in front of them spreads incoming requests across the pool, the data-parallel serving pattern of Chapter 23. The router is not a plain round-robin balancer; because prefix caching makes a replica far cheaper for a request whose prefix it has already cached, a prefix-aware router sends agents sharing a system prompt to the same replica, raising the cache-hit rate across the fleet rather than scattering the shared prefix across every node.

Demand is not constant. Agent traffic is bursty: a batch job or a wave of users fans out into thousands of concurrent calls, then subsides. Autoscaling matches replica count to load, and the right signal is not CPU but the serving queue, the number of requests waiting and the time they wait. The cluster layer of Chapter 33 places each new replica on GPUs that satisfy its tensor-parallel group's topology (all $N$ shards on one node, on the fast interconnect), and scales the pool down when the queue drains. The metric that ties capacity to the service-level objective (SLO) is goodput: the rate of requests served within the latency target, not the raw throughput. Writing $\lambda$ for the offered request rate and $\Pr[\text{latency} \le \text{SLO}]$ for the fraction meeting the budget,

and the fleet is sized so that goodput tracks $\lambda$ across the bursts. Push concurrency too high on a replica and per-token latency rises until requests miss the SLO; goodput then falls even as raw throughput climbs, which is the quantitative signal to add a replica rather than pack the existing ones tighter. This goodput-under-SLO framing is the same one Chapter 24 uses to size LLM fleets, applied here to the agent workload, and it feeds directly into the cost model that Section 40.8 builds next.