Why Model Serving Differs from Web Serving

"They handed me a load balancer that was perfect for web traffic. It spread my requests so evenly that not one of them ever met a friend, and my GPU sat there at four percent, lonely and underfed."

A Serving Replica That Just Wanted a Full Batch

Chapter 22 took a single accelerator and made it as efficient as it could be: quantized weights, a paged key-value cache, fused attention kernels, the per-node craft of getting the most tokens per second out of one box. That work is the foundation, and this part assumes it. But one optimized box is still one box, with a finite memory ceiling, a finite throughput ceiling, and a finite probability of failing at the worst moment. The moment a deployed model must answer more requests than one accelerator can clear, or must stay up while individual machines crash, the problem becomes distributed: a fleet of replicas behind a router, the genuinely distributed serving problem this chapter is about. Chapter 22 built the node; this chapter builds the fleet around it.

The tempting shortcut is to treat that fleet as a solved problem. We already know how to put many identical servers behind a load balancer and serve millions of stateless HTTP requests; surely a model is just another stateless service with a heavier handle_request function. That intuition is wrong in three specific, consequential ways, and naming them is the work of this section. Each one invalidates a design choice that web serving makes almost without thinking, and each one forces a replacement that the rest of the chapter develops in full.

1. The Three Properties That Break Web-Serving Assumptions Beginner

Standard web serving rests on a small set of assumptions that hold so reliably for HTTP traffic that they fade into the background. Requests are independent and stateless, so any replica can serve any request. Replicas are cheap and start in milliseconds, so the autoscaler can add or remove them freely in response to load. Latency is dominated by the work of one request, so spreading requests as thinly as possible across replicas minimizes the time each one waits. A model fleet violates all three. Table 23.1.1 lays the contrast out directly; the three subsections that follow take each row in turn.

1.1 Property one: the GPU wants requests grouped, not spread

An accelerator runs a neural network as a sequence of large matrix operations. Those operations are bandwidth-bound and latency-bound at small sizes: launching a forward pass for a single request pays a large fixed cost (kernel launch, weight reads, pipeline fill) and then does very little useful work. Feed the same pass a batch of requests and the fixed cost is amortized across all of them, while the per-request marginal cost stays tiny. The result is that throughput climbs steeply with batch size before saturating, so the system maximizes throughput by accumulating requests and running them together, the exact opposite of the web instinct to dispatch each request the instant it arrives. This is the latency-throughput tension in its sharpest form: a small, deliberate wait to fill a batch buys a large gain in throughput. We met that tension as a general scaling phenomenon in Section 3.4; here it stops being a curve on a slide and becomes the central routing decision of the whole serving system.

1.2 Property two: replicas are huge and slow to start

A web replica is a few megabytes of code that forks in milliseconds, which is why autoscalers add and remove them freely. A model replica must load weights that are gigabytes to hundreds of gigabytes, pull them across the network or off disk, move them onto the accelerator, and warm up the kernels and the allocator before it can serve a single token. That startup is measured in seconds for a small model and minutes for a large one. You therefore cannot spin up a replica per request, and you cannot let an autoscaler react to a traffic spike by booting cold replicas that arrive after the spike is over. The cost and slowness of a replica forces autoscaling to be predictive and conservative, and forces a pool of pre-warmed replicas to absorb bursts, the subjects of Section 23.4 and Section 23.6.

1.3 Property three: generation is stateful through the KV cache

A classification model returns one answer per request and keeps nothing between requests; it is genuinely stateless, and web-style routing handles it well. Autoregressive generation is different. To produce a sequence token by token, the model keeps a key-value cache, the stored attention keys and values for every token generated so far, on the accelerator, and each new token reads and extends it. That cache, whose per-node economics Chapter 22 dissected in Section 22.5, ties a multi-step generation request to the specific replica that holds it. A stateless load balancer that re-routes the next token of an in-flight generation to a different replica finds no cache there and either fails or silently recomputes the entire history. Statefulness means the router must keep a session on its replica, and means failover has to reckon with lost cache, which Section 23.2 and Section 23.7 address directly.

2. Watching the Wrong Reflex Fail: Spread vs Batch Intermediate

The clearest way to feel the first property is to simulate two routers against the same stream of requests and a simple model of one accelerator. Our accelerator runs a batch of requests in a single forward pass whose time is a fixed overhead plus a small marginal cost per request: a pass over $b$ requests costs $\text{BASE} + \text{PER}\cdot b$ milliseconds but serves all $b$ at once. With $\text{BASE}=8$ ms and $\text{PER}=0.5$ ms, one request alone costs $8.5$ ms (a throughput ceiling near $118$ requests per second), while a full batch of $32$ costs $24$ ms yet clears $32$ requests, more than ten times the work for less than three times the time. The web-style router dispatches each request the instant it arrives, alone, so every pass is a batch of one. The batch-aware router holds arrivals for a short window (here $10$ ms, or until the batch is full at $32$) and then fires one pass for the whole group. The diagram in Figure 23.1.1 contrasts the two before we run them.

The code below implements both routers and the accelerator model, then runs them against two workloads drawn from the same generator: a short overload burst whose offered rate far exceeds one replica's instant-dispatch ceiling, and a light stream whose rate sits well below it. Reporting throughput together with the median (p50) and tail (p99) latency for each makes the tradeoff visible from both sides.

import random

# A GPU runs a BATCH in one forward pass: fixed overhead BASE plus a small
# marginal cost PER per request. A pass over b requests costs BASE + PER*b ms
# but serves all b at once, so 32-in-one-pass beats 32 single-request passes.
BASE, PER, MAX_BATCH = 8.0, 0.5, 32
def pass_time(b): return BASE + PER * b

def web_style(arr):                      # dispatch each request instantly, alone
    end, lat = 0.0, []
    for t in arr:
        start = max(t, end)              # queue behind the previous pass
        end = start + pass_time(1)       # one request per forward pass
        lat.append(end - t)
    return end, lat

WINDOW = 10.0                            # ms the router waits to fill a batch
def batch_aware(arr):                    # accumulate a window, then one pass
    end, lat, i = 0.0, [], 0
    while i < len(arr):
        open_t, j = arr[i], i
        while j < len(arr) and arr[j] <= open_t + WINDOW and (j - i) < MAX_BATCH:
            j += 1                        # admit arrivals within the window
        batch = arr[i:j]
        start = max(batch[-1], end)       # GPU free and window closed
        end = start + pass_time(len(batch))
        lat.extend(end - t for t in batch)
        i = j
    return end, lat

def row(name, mk, lat):
    lat = sorted(lat); n = len(lat)
    print(f"  {name:<12} throughput={1000.0*n/mk:8.1f} req/s   "
          f"p50={lat[n//2]:8.2f} ms   p99={lat[int(0.99*n)]:8.2f} ms")

def scenario(label, span_ms, n=1500, seed=0):
    random.seed(seed)
    arr = sorted(random.uniform(0, span_ms) for _ in range(n))
    print(f"{label}  ({n} requests over {span_ms/1000:.0f} s, "
          f"offered {1000.0*n/span_ms:.0f} req/s):")
    mk_w, lat_w = web_style(arr); mk_b, lat_b = batch_aware(arr)
    row("web-style", mk_w, lat_w); row("batch-aware", mk_b, lat_b)
    print(f"  -> batch-aware clears {(n/mk_b)/(n/mk_w):.1f}x the throughput\n")

scenario("OVERLOAD",   span_ms=3000)     # burst above one replica's ceiling
scenario("LIGHT LOAD", span_ms=20000)    # stream well below the ceiling

OVERLOAD  (1500 requests over 3 s, offered 500 req/s):
  web-style    throughput=   117.6 req/s   p50= 4876.94 ms   p99= 9666.34 ms
  batch-aware  throughput=   497.4 req/s   p50=   17.10 ms   p99=   27.64 ms
  -> batch-aware clears 4.2x the throughput

LIGHT LOAD  (1500 requests over 20 s, offered 75 req/s):
  web-style    throughput=    74.9 req/s   p50=   12.18 ms   p99=   67.27 ms
  batch-aware  throughput=    75.0 req/s   p50=    9.50 ms   p99=   19.12 ms
  -> batch-aware clears 1.0x the throughput

Read the two regimes together, because each tells half the truth. Under the overload burst the web-style reflex is catastrophic: capped at one request per pass, the single replica clears only $118$ requests per second against an offered $500$, so its queue grows without bound and median latency climbs to nearly five seconds. The batch-aware router, running the same hardware, accumulates full batches and clears more than four times the throughput at a median latency of seventeen milliseconds. Under light load, where the accelerator is rarely busy, the two routers are indistinguishable in throughput (both are limited by how fast requests arrive), and the ten-millisecond accumulation window adds no visible cost; the batch-aware policy even tightens the tail by sweeping up the occasional cluster of near-simultaneous arrivals. The lesson is not that batching is always faster but that it is the only policy that does not fall apart under load, and that the small wait it asks for is nearly free when load is light. That asymmetry is exactly why production model servers batch by default and why Section 23.2 makes batch-aware routing the core of the fleet rather than an optimization bolted on later.

3. What This Chapter Builds on Top of the Optimized Node Beginner

The three properties and the experiment above set the agenda for the rest of the chapter, each section repairing one of the broken assumptions for the distributed fleet. Section 23.2 turns the spread-versus-batch finding into real batch-aware routing across many replicas, with load balancing that accounts for batch occupancy and session affinity that keeps a stateful generation on its replica. Section 23.3 separates online serving (latency-critical, one request at a time) from batch serving (throughput-critical, run offline over a corpus), since the right batching window differs by orders of magnitude between them. Section 23.4 builds autoscaling on GPU utilization and queue depth rather than on request count, because a replica running full batches at high utilization should not be scaled the way a busy web server is. Section 23.5 packs multiple models and tenants onto shared accelerators so expensive hardware is not stranded behind one lightly used model. Section 23.6 attacks the cold-start tax head on with warm pools and fast model loading. Section 23.7 makes the fleet survive failures despite the lost-KV-cache problem, and Section 23.8 shows how production frameworks (Triton, Ray Serve, KServe) package all of this so you assemble a fleet instead of building one from scratch.

# config.pbtxt for one model served by Triton
name: "support_llm"
max_batch_size: 32
dynamic_batching {
  max_queue_delay_microseconds: 10000   # wait up to 10 ms to fill a batch
  preferred_batch_size: [ 16, 32 ]      # aim for these efficient batch sizes
}
instance_group [ { count: 2, kind: KIND_GPU } ]   # 2 replicas on GPUs

4. The Frontier: Batching Smarter and Disaggregating the Request Advanced

The fixed-window batcher of Code 23.1.1 is the simplest member of a fast-moving family. Its crudest limitation is that, for generation, a single forward pass is not the whole request: a request runs for many token steps, and a static batch formed once at admission either makes early-finishing requests wait for slow ones or wastes slots as requests complete. The research and systems community has pushed well past it, and the moves below are the ones that Section 23.2 and Chapter 24 build on.

With the three properties named, the wrong reflex measured, and the chapter's agenda set, we are ready to build the fleet for real. The next section takes the spread-versus-batch finding and turns it into routing across many replicas: how to balance load when the unit of work is a batch rather than a request, and how to keep a stateful generation on the replica that holds its cache. That work begins in Section 23.2.