Inference Engines and Practice

"I have read every paper in this chapter. I just call them by their command-line flags now."

An Inference Engine With Too Many Backends

The previous eight sections built distributed LLM serving from its parts. Section 24.2 split one model across devices with tensor parallelism; Section 24.3 stretched it across nodes with pipeline parallelism; Sections 24.4 and 24.5 made the KV cache paged, distributed, and disaggregated from prefill; Section 24.6 scheduled continuous batches across the fleet; Section 24.7 added prefix caching and multi-LoRA; and Section 24.8 served distributed mixture-of-experts (MoE) models with expert parallelism. None of those mechanisms is something you implement from scratch in production. They live inside inference engines, the serving runtimes that turn a model checkpoint and a request stream into tokens at a target latency and throughput. This section is the practitioner's map of that software layer: what the engines share, where they diverge, and how to choose. Figure 24.9.1 assembles the full stack the chapter has built, labeling each layer with the section that produced it, so the engine can be read as the place where those mechanisms are packaged behind a request API.

1. What Every Engine Shares Beginner

The engines converged on a common core because the mechanisms in this chapter are not optional once you serve at scale; they are the only known way to hit competitive throughput and latency together. Every serious engine today does continuous (in-flight) batching rather than static batching, so a finished sequence frees its slot mid-step instead of stalling the batch. Every one pages the KV cache after the PagedAttention design of Chapter 22, so memory fragmentation no longer caps the batch size. Every one supports tensor parallelism within a node and most support pipeline parallelism across nodes, the splits of Chapter 16 applied to inference. And every one now exposes an OpenAI-compatible HTTP endpoint, so the application code above the engine does not change when you swap engines underneath.

This shared core is exactly the set of ideas the chapter built. An engine, in this framing, is a packaging of Chapter 24's mechanisms behind a request API, tuned and kernel-optimized for a particular hardware target. The differences that remain, and they are the ones that decide your choice, sit in four places: how kernels are produced (interpreted versus ahead-of-time compiled), how aggressively prefixes are shared, whether prefill/decode disaggregation is a first-class feature, and which accelerators are supported. We take the three leading open engines in turn.

2. vLLM: The Open Default Beginner

vLLM is the engine most teams reach for first, and for good reason: it introduced PagedAttention, it ships continuous batching, tensor and pipeline parallelism, automatic prefix caching, and multi-LoRA serving (the LoRA fleets of Section 24.7) in one package, and it runs on NVIDIA, AMD, and other backends. Its design philosophy is broad hardware reach and fast iteration in Python with high-performance custom kernels underneath, rather than a heavyweight compile step. For most workloads it delivers strong throughput out of the box with a one-line launch, which is why it has become the de facto open baseline that other engines benchmark against.

# vLLM: tensor parallelism + automatic prefix caching + LoRA adapters
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 8 \        # split the model across 8 GPUs (24.2)
    --enable-prefix-caching \         # share system-prompt prefixes (24.7)
    --enable-lora --max-loras 16 \    # serve 16 LoRA adapters on one base (24.7)
    --max-num-seqs 256                # continuous-batching width (24.6)

The same hardware reach that makes vLLM the safe default also makes it the engine where new research lands first, because contributors can iterate without a compile gate. Prefill/decode disaggregation (Section 24.5) and expert-parallel MoE serving (Section 24.8) both arrived in vLLM as configurable features, which is the practical reason this book's mechanisms are reachable from a single tool.

3. TensorRT-LLM: Compiled Throughput on NVIDIA Intermediate

TensorRT-LLM is NVIDIA's engine, and its bet is compilation. Instead of dispatching kernels at runtime, it builds an optimized engine plan ahead of time: it fuses operations, selects kernels tuned for the exact GPU architecture, and bakes in the chosen precision (including FP8 on Hopper and later). On NVIDIA hardware this regularly yields the highest throughput and lowest latency of the three, because the compiled plan removes per-step dispatch overhead and exploits hardware features that a general runtime cannot assume. The cost is build complexity and rigidity: you compile an engine for a specific model, precision, parallelism degree, and batch profile, and changing any of those means rebuilding. It is the choice when you serve a fixed model at very high volume on NVIDIA GPUs and every percent of throughput is money.

4. SGLang: Prefix Sharing and Structured Generation Intermediate

SGLang's distinctive contribution is RadixAttention, a prefix cache organized as a radix tree so that any shared prefix across requests, not only a fixed system prompt, is detected and reused automatically. This is the Section 24.7 idea pushed to its general form: branching conversations, few-shot templates, and agent traces that share long common stems get near-perfect cache reuse without the application managing it. SGLang pairs this with a frontend language for structured and programmatic generation (constrained decoding, control flow, parallel calls), which makes it especially fast for the templated, agentic, and tool-calling workloads of Part IV's large models when they are driven by the orchestration patterns of Part VI. For traffic dominated by shared prefixes and structured outputs it often leads on throughput; for unstructured, low-overlap traffic the advantage narrows.

5. The Rest of the Field Beginner

Three engines do not exhaust the landscape. Hugging Face's Text Generation Inference (TGI) is a production server tightly integrated with the Hugging Face ecosystem and a common choice when you already live there. LMDeploy, from the OpenMMLab community, is known for strong quantized inference and its TurboMind kernels. DeepSpeed-Inference and the older FasterTransformer contributed kernels and ideas that the newer engines absorbed. And above all of these sit managed endpoints: hosted serving from cloud and model providers that run one of these engines for you behind an API, trading control and cost transparency for the elimination of every operational concern in this chapter. The managed option is the right one whenever your team's comparative advantage is the application, not the serving fleet.

These engines do not replace the generic serving frameworks of Chapter 23; they sit inside them. Ray Serve, KServe, or Triton handle autoscaling, multi-model routing, and rollout, and call an LLM engine as the per-replica runtime. And the engines feed upward into applications: the agentic systems of Part VI and the retrieval-augmented systems of Part VIII are clients of exactly the endpoints this section configures. The engine is one well-defined layer, bounded below by the per-node efficiency of Chapter 22 and above by the orchestration of Chapter 23.

6. A Decision Guide Intermediate

The choice reduces to four questions, ordered by how often they decide the outcome. First, hardware: if you are not on NVIDIA, TensorRT-LLM is out and vLLM's broad backend support leads. Second, stability versus flexibility: a fixed, high-volume model rewards TensorRT-LLM's compiled throughput, while churning models and frequent LoRA swaps reward vLLM's no-compile iteration. Third, traffic shape: heavy shared-prefix and structured/agentic traffic favors SGLang's RadixAttention; unstructured traffic narrows that edge. Fourth, team capacity: when serving is not your differentiator, a managed endpoint or TGI inside an existing stack beats running your own. Table 24.9.1 condenses the comparison; the configurator below turns these axes into a concrete recommendation.

7. A Serving Configurator Intermediate

The decision guide picks an engine; you still have to size the deployment. The throughput of a decode step is set by memory bandwidth, because every generated token reads the model weights once across the shards that hold them. For a model of $P$ parameters in fp16 (2 bytes each) served over $\text{TP}\times\text{PP}$ GPUs of aggregate bandwidth $B$ bytes per second, the per-replica decode rate is approximately

and the tensor-parallel degree is the smallest split that lets the weights plus a context's worth of KV cache fit under a target fraction of each card's memory. The configurator below encodes exactly this reasoning: given model size, context length, SLOs (TTFT and time-per-output-token, TPOT), and a GPU, it recommends a parallelism, disaggregation, and caching plan and estimates throughput and node count. It is the same arithmetic the per-node models of Chapter 22 fed into the fleet-sizing of Chapter 23, now closed into a recommender.

GPU_DB = {  # name: (memory_GB, fp16_TFLOPs, hbm_GB_per_s)
    "A100-80GB": (80, 312, 2039),
    "H100-80GB": (80, 989, 3350),
}

def recommend(name, params_B, ctx_len, ttft_ms, tpot_ms, gpu,
              n_gpu_per_node, qps, kv_bytes_per_tok):
    mem, _tflops, hbm = GPU_DB[gpu]
    w_gb = params_B * 2 * 1.15                       # fp16 weights + overhead
    kv_gb_seq = ctx_len * kv_bytes_per_tok / 1e9     # KV cache for one sequence
    usable = mem * 0.70                              # leave head-room per card
    tp = 1                                           # tensor-parallel degree
    while w_gb / tp + kv_gb_seq > usable and tp < n_gpu_per_node:
        tp *= 2
    pp = 1                                           # pipeline-parallel degree
    while tp * pp < w_gb / usable:
        pp += 1
    disagg = (ctx_len >= 8000) and (ttft_ms <= 500)  # split prefill/decode (24.5)
    prefix_cache = ctx_len >= 2000                   # share long prefixes (24.7)
    tok_s = (hbm * 1e9 * tp * pp) / (params_B * 2 * 1e9)   # decode rate, B/(2P)
    tpot_pred = 1000.0 / tok_s
    req_s_replica = max(tok_s / 200.0, 0.1)          # ~200 gen tok/s per request
    replicas = max(1, -(-int(qps * 100) // int(req_s_replica * 100)))
    per_replica = tp * pp * (2 if disagg else 1)
    total = replicas * per_replica
    nodes = -(-total // n_gpu_per_node)
    print(f"{name}: TP={tp} PP={pp} disagg={'Y' if disagg else 'n'} "
          f"prefix={'Y' if prefix_cache else 'n'} "
          f"TPOT~{tpot_pred:.1f}ms({'ok' if tpot_pred<=tpot_ms else 'MISS'}) "
          f"{tok_s:.0f}tok/s x{replicas}rep -> {total} GPUs / {nodes} nodes")

recommend("13B chat",  13,  2048, 300, 40, "A100-80GB", 8, 20,   800_000)
recommend("70B RAG",   70, 16000, 400, 50, "H100-80GB", 8,  8, 2_500_000)
recommend("405B long", 405,32000, 800, 60, "H100-80GB", 8,  4, 4_000_000)

13B chat: TP=1 PP=1 disagg=n prefix=Y TPOT~12.8ms(ok) 78tok/s x52rep -> 52 GPUs / 7 nodes
70B RAG: TP=8 PP=1 disagg=Y prefix=Y TPOT~5.2ms(ok) 191tok/s x9rep -> 144 GPUs / 18 nodes
405B long: TP=8 PP=3 disagg=n prefix=Y TPOT~10.1ms(ok) 99tok/s x9rep -> 216 GPUs / 27 nodes

The three rows of Output 24.9.2 trace the whole chapter in one table. Replication alone handles a small model; tensor parallelism within a node handles a medium one; pipeline parallelism across nodes plus disaggregation and prefix caching handle the large ones, and every recommendation is just the mechanisms of Sections 24.2 through 24.8 selected by the binding constraint. A real engine refines these numbers with quantization, speculative decoding, and measured kernel timings, but the structure of the decision, which parallelism, whether to disaggregate, where caching pays, is exactly what the configurator captures.

That closes the distributed serving of a single large model. The next chapter turns to the other half of a production LLM system: the retrieval layer that feeds it context. Vector search and approximate nearest-neighbor indices are themselves distributed systems, sharded and replicated like everything else in this book, and Chapter 25 builds them.

8. vLLM V1: The 2025 Architecture Advanced

The vLLM described in Section 2 above is V0, the design introduced with PagedAttention in 2023. In late 2024 the vLLM team released V1, a ground-up rewrite that keeps the same external API and the same PagedAttention KV-cache semantics while replacing the internal execution model. Understanding V1 matters in practice because it is the default vLLM as of 2025, and its performance characteristics differ substantially from the V0 baseline that most benchmarks cite.

What Changed from V0 to V1

V0 used a synchronous executor loop: the scheduler, the model runner, and the KV-cache manager shared state in a single Python process. On each step the scheduler decided which requests to advance, invoked the model runner for a forward pass, then updated the cache. Under low concurrency this was fine. Under high concurrency the scheduler blocked on the model forward pass, which meant that scheduling decisions for the next batch could not begin until the current GPU kernel finished. The GIL serialized everything, and the single-process design offered no way to overlap scheduling with execution.

V1 decouples these responsibilities into three cooperating asynchronous components. An async Scheduler is responsible only for deciding which requests to advance and which KV blocks to allocate; it never waits for GPU execution. An async Engine Core owns model execution and pushes completed token logits into an output queue as soon as the forward pass returns. Per-client output processors pull from that queue and handle detokenization independently of the main loop. The three components communicate by queue-based message passing: the Engine Core pushes finished logits into an output queue; the Scheduler pulls request-status updates from a status queue. Neither component holds the other's lock during execution. This eliminates the GIL bottleneck that serialized V0's step loop and allows the scheduler to prepare the next batch while the GPU is executing the current one.

Key V1 Features

Beyond the async pipeline, V1 ships four mechanisms that were optional or absent in V0. First, prefix caching is enabled by default: automatic KV-cache reuse for any repeated prefix, including system prompts, few-shot templates, and branching agent traces, requires no configuration flag. Second, native FP8 support on Hopper (H100) GPUs uses PyTorch's fp8 dtype rather than a custom kernel, making quantized serving a first-class path. Third, chunked prefill is the default scheduler policy: long prefills are split into fixed-size chunks (see Section 24.6 and Chapter 22's chunked-prefill discussion), so decode batches are never starved by a large prompt. Fourth, a modular executor abstraction exposes a uniform interface for CPU, single-GPU, multi-GPU tensor-parallel, and multi-node pipeline-parallel backends, making it easier to add new accelerators without rewriting the engine core.

Performance

On LLaMA-3-8B under high-concurrency workloads (128 or more concurrent requests), V1 achieves 1.4 to 2.1 times the throughput of V0 with identical hardware. The gain is larger when prompts are long (the chunked prefill default eliminates head-of-line blocking) and when many requests share a system prompt (prefix caching eliminates redundant KV computation). The gains are smaller for short, low-overlap traffic where the scheduling overhead of V0 was already negligible.