Serving Distributed MoE Models

"They batched a thousand requests together just so two of them would finally have enough company to make me worth waking up."

An Expert Idling Below Tile Size

The earlier chapters of this part sized a dense serving deployment from per-node economics. Chapter 22 established the per-GPU baseline (quantization, paged KV cache, the memory budget a single accelerator brings), and Chapter 23 multiplied that baseline across a fleet. A dense model serves predictably: its FLOPs per token are fixed, so more requests means more GPUs in a clean linear way. A sparse MoE model breaks that clean accounting on two axes at once. The fleet is sized not by arithmetic but by total parameters, because all experts must be loadable; and the latency per token is set not by compute but by a collective, the inference all-to-all from Chapter 4, which returns here as the dominant cost at exactly the batch sizes decode runs at. This is the same all-to-all that the thesis thread of this book has tracked from a hand-written sum in Chapter 1 to expert routing in Chapter 17; at serving time it surfaces as the thing you optimize a deployment around.

1. The Serving Paradox, Restated for Decode Beginner

Section 17.8 stated the MoE serving paradox in its general form: cheap per token, heavy in total. With $E$ experts and top-$k$ routing, a token's feed-forward block uses only $k$ of the $E$ experts, so the active parameter count per token is a fraction $k/E$ of the layer's total. But routing is data-dependent and decided at inference time, so the system must be ready to send any token to any expert. Every expert is therefore loaded into device memory somewhere on the fleet, all the time. Resident weight memory scales with the full expert count while compute scales only with the active experts, and the ratio of what you provision to what a token consumes is

where $b$ is bytes per parameter and $F_{\text{token}}$ the per-token FLOPs. A model with $E = 64$ and $k = 2$ keeps thirty-two times more parameters resident than any single token pays compute for, so the dense habit of sizing a fleet from FLOPs is off by that factor. Serving is sized by the memory wall first.

What decode adds to this picture is a second, sharper problem that training never faced. Training runs at enormous batch sizes: tens of thousands of tokens flow through each MoE layer per step, so every expert receives a healthy crowd of tokens, fills its GEMM tiles, and amortizes the all-to-all over a large message. Autoregressive decode is the opposite. Each in-flight request contributes exactly one token per step, so the per-layer batch equals the number of concurrent requests, often a few dozen. Spread $k$ times that handful of tokens across $E$ experts and most experts receive a fraction of a token's worth of work. The expert kernels run far below their efficient tile size, and the all-to-all ships tiny messages dominated by fixed per-hop latency. The sparsity that was pure profit at training scale becomes a structural inefficiency at decode scale. We make the cost of distribution explicit, as Chapter 3 taught, so the trade-off can be reasoned about with numbers rather than instinct.

2. Experts Resident, Few Tokens in Flight Beginner

Figure 24.8.1 shows the decode-time arrangement and why it is awkward. The experts of each MoE layer are sharded across the serving GPUs in an expert-parallel layout, every GPU permanently holding its slice of experts, exactly the inference-time face of the expert parallelism from Chapter 17. When a small decode batch arrives, the router on each GPU picks $k$ experts per token, the first all-to-all ships each token to the GPU owning its chosen expert, the experts compute, and a second all-to-all returns the results. The experts are stationary; the tokens travel. At decode batch sizes the picture is stark: a handful of tokens scatters across a wide fleet of experts, so most experts compute on one or two tokens or sit idle this step, while the all-to-all links carry near-empty messages whose cost is almost entirely the latency of setting up the transfer.

3. Modeling Decode Efficiency as Batch Size Grows Intermediate

The cleanest way to see the all-to-all bound and its cure is to put numbers to it. We model one MoE layer at decode. With batch size $B$ (the number of concurrent decode requests, one token each) and top-$k$ routing across $E$ experts, the mean tokens landing on an expert is

The per-layer all-to-all costs a fixed latency $\alpha$ plus a bandwidth term proportional to the bytes it moves, $t_{\text{a2a}} = \alpha + t_{\text{expert}} \cdot H \cdot b / \beta$, where $H$ is the hidden size and $\beta$ the link bandwidth. The expert GEMM pays a fixed weight-load floor plus per-token arithmetic that only reaches peak once a tile of $\tau$ tokens is full. Time-per-output-token is the per-layer maximum of these two terms (the all-to-all overlaps the next micro-step's GEMM) summed over the $L$ MoE layers. The code below evaluates this model across decode batch sizes and reports, for each, the mean tokens per expert, the GEMM tile efficiency, the share of per-layer time spent in the all-to-all, and the resulting TPOT.

import math

E, k, L = 64, 2, 58          # experts, top-k, MoE layers (DeepSeek-V3-scale)
TILE, HIDDEN, BYTES = 128, 7168, 2          # GEMM tile, hidden size, bf16
A2A_ALPHA_US, A2A_GBPS = 8.0, 400.0         # fixed all-to-all latency, link GB/s
GEMM_PEAK_US, GEMM_FIXED_US = 8.0, 1.0      # full-tile GEMM time, weight-load floor

def model(B):
    tpe = B * k / E                                   # mean tokens per expert
    a2a_us = A2A_ALPHA_US + (tpe * HIDDEN * BYTES) / (A2A_GBPS * 1e3)   # latency + bytes/bw
    per_tok_us = GEMM_PEAK_US / TILE                  # peak per-token GEMM cost
    gemm_us = GEMM_FIXED_US + per_tok_us * tpe        # weight load + arithmetic
    gemm_eff = (per_tok_us * tpe) / gemm_us if gemm_us else 0.0   # fraction doing real work
    layer_us = max(a2a_us, gemm_us)                   # a2a overlaps next GEMM
    a2a_share = a2a_us / (a2a_us + gemm_us)
    return tpe, gemm_eff, a2a_share, layer_us * L / 1000.0        # TPOT in ms

print(f"{'batch':>6} {'tok/expert':>11} {'gemm_eff':>9} {'a2a_share':>10} {'TPOT_ms':>9}  regime")
for B in [1, 4, 16, 64, 256, 1024, 4096]:
    tpe, eff, a2a, tpot = model(B)
    regime = "all-to-all-bound" if a2a >= 0.5 else "compute-bound"
    print(f"{B:>6} {tpe:>11.2f} {eff:>9.2f} {a2a:>10.2f} {tpot:>9.2f}  {regime}")

print()
print(f"batch for >=90% expert-GEMM tile efficiency : {TILE * 0.9 * E / k:.0f} concurrent decodes")
print(f"E/k memory-vs-compute provisioning ratio     : {E // k}x")

 batch  tok/expert  gemm_eff  a2a_share   TPOT_ms  regime
     1        0.03      0.00       0.89      0.46  all-to-all-bound
     4        0.12      0.01       0.89      0.46  all-to-all-bound
    16        0.50      0.03       0.89      0.47  all-to-all-bound
    64        2.00      0.11       0.88      0.47  all-to-all-bound
   256        8.00      0.33       0.85      0.48  all-to-all-bound
  1024       32.00      0.67       0.75      0.53  all-to-all-bound
  4096      128.00      0.89       0.58      0.73  all-to-all-bound

batch for >=90% expert-GEMM tile efficiency : 3686 concurrent decodes
E/k memory-vs-compute provisioning ratio     : 32x

The numbers tell the whole story. At the batch sizes that interactive decode actually runs at, the experts are starved and the all-to-all latency, repeated across all $L = 58$ MoE layers, owns the step. Efficiency does not arrive until the batch is enormous, far larger than any single decode request stream produces, and even at $B = 4096$ the all-to-all still claims well over half of each layer. Two levers follow directly from this model: make the effective batch per expert larger (so tiles fill and the latency amortizes), or make the all-to-all cheaper and more hidden (so its fixed cost stops dominating). The rest of the section is those two levers in practice. Notice also the final two lines: the $32\times$ memory provisioning ratio from Section 1 and the batch needed for tile efficiency are the two walls, memory and batching, that define an MoE serving deployment.

4. Lever One: Batching and Hot-Expert Replication Intermediate

The first lever is to grow the tokens each expert sees. Continuous batching, the cross-node request scheduling that Chapter 23 built for dense serving, helps here too: by packing many in-flight decode requests into one step, it raises $B$ and therefore $t_{\text{expert}}$ for free. But there is a ceiling. Output 24.8.1 shows you need thousands of concurrent decodes to fill tiles, which a single deployment may never reach, and pushing batch size up trades latency for throughput, which a strict TPOT budget forbids. Batching alone closes part of the gap, not all of it.

Hot-expert replication attacks the imbalance directly. Real routing is not uniform: a handful of experts attract a disproportionate share of tokens (the load-imbalance problem of Chapter 17), and at small batch a single overloaded expert stretches the whole layer while its peers idle. Replicating the busiest experts onto additional GPUs lets the router spread a hot expert's tokens across several replicas, so no one expert becomes the layer's straggler. The memory cost is real, you pay for the duplicated experts, but it is far smaller than replicating the whole model, and it directly cuts the tail latency that load imbalance creates at decode. Placement matters too: co-locating experts that are frequently co-activated on the same GPU or NVLink island shortens the all-to-all paths, the topology-aware placement idea from Chapter 4 applied to experts instead of gradients.

5. Lever Two: Overlapping and Shrinking the All-to-All Advanced

The second lever accepts that the all-to-all will move data on every layer and works to make that movement nearly free. The first technique is overlap: while the experts on a GPU compute the current micro-batch, the network simultaneously ships the next micro-batch's tokens, so the all-to-all latency hides behind the attention and expert compute rather than adding to it. This is the same computation-communication overlap that Chapter 4 introduced for collectives, now scheduled tightly enough that the all-to-all nearly disappears behind the work it serves. Dedicated MoE communication libraries push this furthest, fusing the dispatch and combine all-to-alls with the expert GEMMs and using high-throughput GPU-initiated transfers so the network is busy whenever compute is.

The second technique is to shrink the bytes the all-to-all moves: dispatch tokens in a low-precision format (the hidden vectors crossing the network in FP8 rather than bf16 halve the message), and fuse the two all-to-alls so routing metadata is exchanged once. When even residency is impossible, expert offloading and caching trade latency for memory: cold experts live in CPU memory or are streamed from a fast tier, and only the experts a batch actually routes to are pulled onto the GPU, with a cache keeping recently hot experts resident. This makes a model whose full expert set exceeds GPU memory servable on a smaller fleet, at the cost of a fetch latency that prefetching and good placement try to hide. The frontier deployments combine all of these, which is why a model at the scale of DeepSeek-V3 is served on a large expert-parallel cluster rather than a handful of GPUs.

# vLLM: serve a large MoE with expert parallelism across 8 GPUs
vllm serve deepseek-ai/DeepSeek-V3 \
  --tensor-parallel-size 1 \
  --enable-expert-parallel \
  --data-parallel-size 8           # experts sharded across the 8-way EP group

# SGLang: the same model with expert parallelism and EP load balancing
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3 \
  --tp 8 --enable-ep-moe           # expert-parallel MoE execution + balanced dispatch

6. Why Frontier MoE Needs a Large Serving Fleet Intermediate

Putting the two walls together explains the deployment shape of frontier MoE. The memory wall sets a floor on fleet size: a model with hundreds of billions of total parameters across hundreds of experts cannot fit on a few GPUs, so the experts must be sharded across many, which is wide expert parallelism by necessity, not choice. The batching wall sets a floor on traffic: to keep those many experts fed at decode, the fleet needs enough aggregate request volume that, after the all-to-all spreads tokens across the expert shards, each expert still sees a worthwhile batch. A large expert-parallel deployment satisfies both at once, many GPUs to hold the experts and enough multiplexed traffic to keep tiles full, which is why a DeepSeek-V3-scale model is served on a sizable cluster with specialized all-to-all kernels rather than squeezed onto a single node. The communication that Chapter 4 taught and Chapter 17 wired into training is, at this scale, the thing the whole serving design is organized around.

The next section steps back from MoE specifically to the inference engines that implement all of this, vLLM, TensorRT-LLM, and SGLang, and how the techniques of this chapter (tensor and pipeline parallelism, paged and disaggregated KV cache, continuous batching, and expert-parallel MoE) compose inside a production serving stack. That tour, the practical close of the chapter, begins in Section 24.9.