Inference Engines and Practice

"I have read every paper in this chapter. I just call them by their command-line flags now."

An Inference Engine With Too Many Backends

The previous eight sections built distributed LLM serving from its parts. Section 24.2 split one model across devices with tensor parallelism; Section 24.3 stretched it across nodes with pipeline parallelism; Sections 24.4 and 24.5 made the KV cache paged, distributed, and disaggregated from prefill; Section 24.6 scheduled continuous batches across the fleet; Section 24.7 added prefix caching and multi-LoRA; and Section 24.8 served distributed mixture-of-experts (MoE) models with expert parallelism. None of those mechanisms is something you implement from scratch in production. They live inside inference engines, the serving runtimes that turn a model checkpoint and a request stream into tokens at a target latency and throughput. This section is the practitioner's map of that software layer: what the engines share, where they diverge, and how to choose. Figure 24.9.1 assembles the full stack the chapter has built, labeling each layer with the section that produced it, so the engine can be read as the place where those mechanisms are packaged behind a request API.

1. What Every Engine Shares Beginner

The engines converged on a common core because the mechanisms in this chapter are not optional once you serve at scale; they are the only known way to hit competitive throughput and latency together. Every serious engine today does continuous (in-flight) batching rather than static batching, so a finished sequence frees its slot mid-step instead of stalling the batch. Every one pages the KV cache after the PagedAttention design of Chapter 22, so memory fragmentation no longer caps the batch size. Every one supports tensor parallelism within a node and most support pipeline parallelism across nodes, the splits of Chapter 16 applied to inference. And every one now exposes an OpenAI-compatible HTTP endpoint, so the application code above the engine does not change when you swap engines underneath.

This shared core is exactly the set of ideas the chapter built. An engine, in this framing, is a packaging of Chapter 24's mechanisms behind a request API, tuned and kernel-optimized for a particular hardware target. The differences that remain, and they are the ones that decide your choice, sit in four places: how kernels are produced (interpreted versus ahead-of-time compiled), how aggressively prefixes are shared, whether prefill/decode disaggregation is a first-class feature, and which accelerators are supported. We take the three leading open engines in turn.

2. vLLM: The Open Default Beginner

vLLM is the engine most teams reach for first, and for good reason: it introduced PagedAttention, it ships continuous batching, tensor and pipeline parallelism, automatic prefix caching, and multi-LoRA serving (the LoRA fleets of Section 24.7) in one package, and it runs on NVIDIA, AMD, and other backends. Its design philosophy is broad hardware reach and fast iteration in Python with high-performance custom kernels underneath, rather than a heavyweight compile step. For most workloads it delivers strong throughput out of the box with a one-line launch, which is why it has become the de facto open baseline that other engines benchmark against.

# vLLM: tensor parallelism + automatic prefix caching + LoRA adapters
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 8 \        # split the model across 8 GPUs (24.2)
    --enable-prefix-caching \         # share system-prompt prefixes (24.7)
    --enable-lora --max-loras 16 \    # serve 16 LoRA adapters on one base (24.7)
    --max-num-seqs 256                # continuous-batching width (24.6)

The same hardware reach that makes vLLM the safe default also makes it the engine where new research lands first, because contributors can iterate without a compile gate. Prefill/decode disaggregation (Section 24.5) and expert-parallel MoE serving (Section 24.8) both arrived in vLLM as configurable features, which is the practical reason this book's mechanisms are reachable from a single tool.

3. TensorRT-LLM: Compiled Throughput on NVIDIA Intermediate

TensorRT-LLM is NVIDIA's engine, and its bet is compilation. Instead of dispatching kernels at runtime, it builds an optimized engine plan ahead of time: it fuses operations, selects kernels tuned for the exact GPU architecture, and bakes in the chosen precision (including FP8 on Hopper and later). On NVIDIA hardware this regularly yields the highest throughput and lowest latency of the three, because the compiled plan removes per-step dispatch overhead and exploits hardware features that a general runtime cannot assume. The cost is build complexity and rigidity: you compile an engine for a specific model, precision, parallelism degree, and batch profile, and changing any of those means rebuilding. It is the choice when you serve a fixed model at very high volume on NVIDIA GPUs and every percent of throughput is money.

4. SGLang: Prefix Sharing and Structured Generation Intermediate

SGLang's distinctive contribution is RadixAttention, a prefix cache organized as a radix tree so that any shared prefix across requests, not only a fixed system prompt, is detected and reused automatically. This is the Section 24.7 idea pushed to its general form: branching conversations, few-shot templates, and agent traces that share long common stems get near-perfect cache reuse without the application managing it. SGLang pairs this with a frontend language for structured and programmatic generation (constrained decoding, control flow, parallel calls), which makes it especially fast for the templated, agentic, and tool-calling workloads of Part IV's large models when they are driven by the orchestration patterns of Part VI. For traffic dominated by shared prefixes and structured outputs it often leads on throughput; for unstructured, low-overlap traffic the advantage narrows.

5. The Rest of the Field Beginner

Three engines do not exhaust the landscape. Hugging Face's Text Generation Inference (TGI) is a production server tightly integrated with the Hugging Face ecosystem and a common choice when you already live there. LMDeploy, from the OpenMMLab community, is known for strong quantized inference and its TurboMind kernels. DeepSpeed-Inference and the older FasterTransformer contributed kernels and ideas that the newer engines absorbed. And above all of these sit managed endpoints: hosted serving from cloud and model providers that run one of these engines for you behind an API, trading control and cost transparency for the elimination of every operational concern in this chapter. The managed option is the right one whenever your team's comparative advantage is the application, not the serving fleet.

These engines do not replace the generic serving frameworks of Chapter 23; they sit inside them. Ray Serve, KServe, or Triton handle autoscaling, multi-model routing, and rollout, and call an LLM engine as the per-replica runtime. And the engines feed upward into applications: the agentic systems of Part VI and the retrieval-augmented systems of Part VIII are clients of exactly the endpoints this section configures. The engine is one well-defined layer, bounded below by the per-node efficiency of Chapter 22 and above by the orchestration of Chapter 23.

6. A Decision Guide Intermediate

The choice reduces to four questions, ordered by how often they decide the outcome. First, hardware: if you are not on NVIDIA, TensorRT-LLM is out and vLLM's broad backend support leads. Second, stability versus flexibility: a fixed, high-volume model rewards TensorRT-LLM's compiled throughput, while churning models and frequent LoRA swaps reward vLLM's no-compile iteration. Third, traffic shape: heavy shared-prefix and structured/agentic traffic favors SGLang's RadixAttention; unstructured traffic narrows that edge. Fourth, team capacity: when serving is not your differentiator, a managed endpoint or TGI inside an existing stack beats running your own. Table 24.9.1 condenses the comparison; the configurator below turns these axes into a concrete recommendation.

7. A Serving Configurator Intermediate

The decision guide picks an engine; you still have to size the deployment. The throughput of a decode step is set by memory bandwidth, because every generated token reads the model weights once across the shards that hold them. For a model of $P$ parameters in fp16 (2 bytes each) served over $\text{TP}\times\text{PP}$ GPUs of aggregate bandwidth $B$ bytes per second, the per-replica decode rate is approximately

and the tensor-parallel degree is the smallest split that lets the weights plus a context's worth of KV cache fit under a target fraction of each card's memory. The configurator below encodes exactly this reasoning: given model size, context length, SLOs (TTFT and time-per-output-token, TPOT), and a GPU, it recommends a parallelism, disaggregation, and caching plan and estimates throughput and node count. It is the same arithmetic the per-node models of Chapter 22 fed into the fleet-sizing of Chapter 23, now closed into a recommender.

GPU_DB = {  # name: (memory_GB, fp16_TFLOPs, hbm_GB_per_s)
    "A100-80GB": (80, 312, 2039),
    "H100-80GB": (80, 989, 3350),
}

def recommend(name, params_B, ctx_len, ttft_ms, tpot_ms, gpu,
              n_gpu_per_node, qps, kv_bytes_per_tok):
    mem, _tflops, hbm = GPU_DB[gpu]
    w_gb = params_B * 2 * 1.15                       # fp16 weights + overhead
    kv_gb_seq = ctx_len * kv_bytes_per_tok / 1e9     # KV cache for one sequence
    usable = mem * 0.70                              # leave head-room per card
    tp = 1                                           # tensor-parallel degree
    while w_gb / tp + kv_gb_seq > usable and tp < n_gpu_per_node:
        tp *= 2
    pp = 1                                           # pipeline-parallel degree
    while tp * pp < w_gb / usable:
        pp += 1
    disagg = (ctx_len >= 8000) and (ttft_ms <= 500)  # split prefill/decode (24.5)
    prefix_cache = ctx_len >= 2000                   # share long prefixes (24.7)
    tok_s = (hbm * 1e9 * tp * pp) / (params_B * 2 * 1e9)   # decode rate, B/(2P)
    tpot_pred = 1000.0 / tok_s
    req_s_replica = max(tok_s / 200.0, 0.1)          # ~200 gen tok/s per request
    replicas = max(1, -(-int(qps * 100) // int(req_s_replica * 100)))
    per_replica = tp * pp * (2 if disagg else 1)
    total = replicas * per_replica
    nodes = -(-total // n_gpu_per_node)
    print(f"{name}: TP={tp} PP={pp} disagg={'Y' if disagg else 'n'} "
          f"prefix={'Y' if prefix_cache else 'n'} "
          f"TPOT~{tpot_pred:.1f}ms({'ok' if tpot_pred<=tpot_ms else 'MISS'}) "
          f"{tok_s:.0f}tok/s x{replicas}rep -> {total} GPUs / {nodes} nodes")

recommend("13B chat",  13,  2048, 300, 40, "A100-80GB", 8, 20,   800_000)
recommend("70B RAG",   70, 16000, 400, 50, "H100-80GB", 8,  8, 2_500_000)
recommend("405B long", 405,32000, 800, 60, "H100-80GB", 8,  4, 4_000_000)

13B chat: TP=1 PP=1 disagg=n prefix=Y TPOT~12.8ms(ok) 78tok/s x52rep -> 52 GPUs / 7 nodes
70B RAG: TP=8 PP=1 disagg=Y prefix=Y TPOT~5.2ms(ok) 191tok/s x9rep -> 144 GPUs / 18 nodes
405B long: TP=8 PP=3 disagg=n prefix=Y TPOT~10.1ms(ok) 99tok/s x9rep -> 216 GPUs / 27 nodes

The three rows of Output 24.9.2 trace the whole chapter in one table. Replication alone handles a small model; tensor parallelism within a node handles a medium one; pipeline parallelism across nodes plus disaggregation and prefix caching handle the large ones, and every recommendation is just the mechanisms of Sections 24.2 through 24.8 selected by the binding constraint. A real engine refines these numbers with quantization, speculative decoding, and measured kernel timings, but the structure of the decision, which parallelism, whether to disaggregate, where caching pays, is exactly what the configurator captures.

That closes the distributed serving of a single large model. The next chapter turns to the other half of a production LLM system: the retrieval layer that feeds it context. Vector search and approximate nearest-neighbor indices are themselves distributed systems, sharded and replicated like everything else in this book, and Chapter 25 builds them.