Pipeline-Parallel and Multi-Node Inference

"I hold layers forty through sixty. I do not know what the prompt was, I do not know what the answer will be, and I refuse to start until the node before me sends my activations. We call this teamwork."

A Pipeline Stage Waiting on Its Upstream

In the previous section the model was split so that every matrix multiply inside every transformer layer ran across several GPUs at once, with the partial results stitched together by an all-reduce on each layer. That tensor-parallel scheme is wonderful inside one node, where the GPUs share an NVLink fabric that moves activations at hundreds of gigabytes per second. It stops being wonderful the moment the all-reduce has to cross a slower network link between nodes, because tensor parallelism communicates on every layer, twice, and a single forward pass through a large model has dozens of layers. So in practice you cap tensor parallelism at the GPUs of one node and ask a different question: if eight GPUs in one box still cannot hold the model, how do you spill it onto a second box without paying the tensor-parallel communication tax across the network?

The answer is to communicate far less often. A transformer is a stack of identical layers, and the only thing layer $i$ needs from layer $i-1$ is that layer's output activations, a single tensor. If you put layers $1$ through $L/S$ on node one, layers $L/S+1$ through $2L/S$ on node two, and so on across $S$ stages, then the entire inter-node traffic for one token is $S-1$ small activation tensors, one per stage boundary, instead of the dozens of all-reduces tensor parallelism would demand. This is pipeline parallelism, and it is the same construction the training chapters built; Section 16.3 developed it for the backward pass of training. Here we re-derive its economics for the forward-only, one-token-at-a-time world of inference, where the bubble has a different shape.

1. Staging Layers Across Nodes Beginner

Pipeline parallelism rests on a structural fact about transformers: they are a deep stack of layers applied in sequence, where each layer consumes only the previous layer's output. That sequential dependency is exactly what lets you cut the stack at any layer boundary and place the two halves on different machines. Cut the $L$-layer model into $S$ contiguous stages, give stage $s$ to node $s$, and a forward pass becomes a relay: node one runs its layers, hands the resulting activation tensor to node two, node two runs its layers, and so on until the final stage produces the output logits. The only thing that ever travels between nodes is that activation tensor, whose size is the hidden dimension times the number of tokens in flight, typically a few megabytes, sent once per boundary rather than the dozens of collective operations tensor parallelism performs.

This is why the two parallelism axes pair so naturally. Tensor parallelism is communication-heavy but the communication is fast inside a node; pipeline parallelism is communication-light, so its communication survives the slow trip between nodes. The standard recipe, visible in Figure 24.3.1, is tensor-parallel within a node and pipeline-parallel across nodes. You scale tensor parallelism up to the GPU count of one box, where NVLink keeps the per-layer all-reduce cheap, and only then reach for pipeline parallelism to span boxes, because its sparse activation passing is the only kind of inter-node traffic that does not strangle throughput.

2. The Inference Bubble Is Not the Training Bubble Intermediate

Splitting layers across stages creates idle time. When the very first token enters the pipeline, only stage one is busy; stages two through $S$ sit idle waiting for activations to reach them. Symmetrically, as the work drains out, the early stages go idle. This idle region is the pipeline bubble, and in training it is fought with micro-batching: a large training batch is chopped into many micro-batches that march through the stages one after another, so that once the pipeline is full every stage is working on a different micro-batch at once. That same trick was the subject of Section 16.3, where micro-batches existed naturally because a training step processes a big batch of examples.

Inference decoding is different, and the difference is the whole point of this section. Autoregressive decoding produces one token at a time: the model must emit token $t$ before it can begin token $t+1$, because token $t+1$ depends on it. A single request therefore offers no micro-batches to fill the pipeline; it is one token walking the relay alone, leaving $S-1$ stages idle at every moment. The bubble does not shrink with longer generation, because the dependency is along the sequence, not across it. The only way to fill the stages is to put different requests in flight at once: while stage two works on request A's token, stage one starts request B's token. The micro-batches of inference are concurrent requests, and the pipeline fills exactly when the serving system has enough simultaneous traffic to keep every stage busy.

Let us make the economics quantitative. Suppose each stage takes $t_{\text{stage}}$ milliseconds to run its layers for one micro-batch, and passing activations across a boundary costs $t_{\text{link}}$ milliseconds. Filling the pipeline costs a one-time bubble of $(S-1)(t_{\text{stage}} + t_{\text{link}})$. After that, the pipeline emits one finished micro-batch every $t_{\text{stage}} + t_{\text{link}}$, so processing $m$ concurrent requests in one decode step takes

and the throughput in tokens per second is $\text{thr}(m) = 1000\, m / T(m)$. As $m$ grows, the fixed bubble is amortized over more requests and throughput climbs toward the ceiling $1000 / (t_{\text{stage}} + t_{\text{link}})$. Crucially, the per-request latency does not improve with $m$: a single token still has to cross all $S$ stages, so its latency is roughly $S\,(t_{\text{stage}} + t_{\text{link}})$ no matter how full the pipe is. Batching buys throughput, never latency, which is the central trade-off of multi-node serving.

3. A Runnable Model of the Filling Pipeline Intermediate

The code below turns the formula above into a small simulator. It sweeps the number of concurrent requests $m$ from one to sixty-four and reports, for a four-stage pipeline, how long each decode step takes, the resulting throughput, and the stage utilization (the fraction of stage-time actually doing work rather than waiting in the bubble). It then isolates the cross-node penalty by recomputing throughput with the link cost set to zero, the single-node fantasy, and comparing it to the real networked number.

t_stage = 6.0     # ms: one stage runs its layers for one micro-batch
t_link  = 2.0     # ms: cross-node activation transfer between adjacent stages
S = 4             # pipeline stages, one per node
warmup = (S - 1) * (t_stage + t_link)   # bubble: time to fill the pipeline

def decode_step(m):                      # m = concurrent requests (micro-batches)
    period   = t_stage + t_link          # a stage emits one micro-batch per period
    total_ms = warmup + m * period       # fill the pipe, then stream m results
    throughput = m / total_ms * 1000.0   # tokens/sec across the whole step
    util = (m * t_stage) / total_ms      # fraction of stage-time doing real work
    return total_ms, throughput, util

print(f"S={S} stages  t_stage={t_stage}ms  t_link={t_link}ms  fill bubble={warmup:.1f}ms")
print(f"{'requests m':>10} {'step ms':>9} {'tok/s':>8} {'util %':>7}")
for m in [1, 2, 4, 8, 16, 32, 64]:
    total_ms, thr, util = decode_step(m)
    print(f"{m:>10} {total_ms:>9.1f} {thr:>8.1f} {util*100:>6.1f}")

# Cross-node penalty: same work with links free (one node) vs the real network.
def thr_at(m, link):
    bubble = (S - 1) * (t_stage + link)
    return m / (bubble + m * (t_stage + link)) * 1000.0
m = 32
print(f"\nat m={m}: links=0ms -> {thr_at(m,0.0):.1f} tok/s, "
      f"links={t_link}ms -> {thr_at(m,t_link):.1f} tok/s, "
      f"penalty {100*(1-thr_at(m,t_link)/thr_at(m,0.0)):.1f}%")

S=4 stages  t_stage=6.0ms  t_link=2.0ms  fill bubble=24.0ms
requests m   step ms    tok/s  util %
         1      32.0     31.2   18.8
         2      40.0     50.0   30.0
         4      56.0     71.4   42.9
         8      88.0     90.9   54.5
        16     152.0    105.3   63.2
        32     280.0    114.3   68.6
        64     536.0    119.4   71.6

at m=32: links=0ms -> 152.4 tok/s, links=2.0ms -> 114.3 tok/s, penalty 25.0%

Three readings repay attention. First, at one concurrent request the pipeline is barely a fifth utilized, and the only cure is more simultaneous traffic; this is the inference bubble made of empty stages. Second, throughput rises steeply at first and then flattens as it approaches the ceiling, so the marginal benefit of yet more concurrency falls off, and there is a sweet spot past which you are buying latency (longer steps) for very little throughput. Third, the cross-node penalty is real and quantified: at thirty-two requests the activation hops shave a quarter off the throughput that an imaginary single-node version would reach. That penalty is the price of admission for serving a model that simply does not fit in one box, and it is why you reach for pipeline parallelism only when forced, never for elegance.

4. Composing With Replication and Routing Intermediate

A single pipeline-by-tensor parallel group can serve only so many requests per second, and it cannot survive a node failure on its own; lose one stage's node and the whole relay breaks, because every request must traverse every stage. Both limits are answered by the layer above, the replication and routing of Chapter 23. You stand up several identical copies of the whole multi-node group, each a complete serving unit, and put a router in front that fans incoming requests across the replicas. This is the data-parallel axis of Figure 24.3.1, stacked on top of the tensor and pipeline axes, and it is what carries the three-dimensional layout from a single oversized model into a fault-tolerant fleet.

The composition is clean because the axes answer different questions. Tensor and pipeline parallelism exist to make one copy of the model fit and run; they are about capacity. Data-parallel replication exists to multiply throughput and to provide spares when a node dies; it is about scale and reliability. The router does not care that each replica is internally a four-node pipeline, it sees an opaque endpoint that accepts a request and returns tokens. Continuous batching, the policy of admitting new requests into a running batch as old ones finish, then operates inside each replica to keep its pipeline filled toward the high-utilization end of Output 24.3.1, and we develop that scheduling in Section 24.6.

from vllm import LLM   # launched across nodes via Ray; vLLM discovers the cluster

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=8,     # split each layer across the 8 GPUs in ONE node
    pipeline_parallel_size=2,   # stage the layers across 2 nodes (a 16-GPU group)
)
# One serving unit = 2 nodes; replicate this unit behind a router (Chapter 23)
# to scale throughput and tolerate node failure.
out = llm.generate(["Explain pipeline parallelism in one sentence."])
print(out[0].outputs[0].text)

5. When to Cross a Node, and When to Refuse Advanced

The discipline this section teaches is restraint. Every node boundary a token must cross adds a network hop to its latency and a slice off its throughput, as Output 24.3.1 made concrete with its 25% penalty. So the rule is to add pipeline stages only when a hard ceiling forces it: the weights plus a usable KV cache do not fit in one node even after tensor parallelism has used every GPU in the box. If the model does fit one node, a pure tensor-parallel single-node deployment replicated by the router will almost always beat a multi-node pipeline on latency, because it pays no inter-node activation tax at all. Multi-node serving is the remedy for a capacity ceiling, not a default.

When you are forced across nodes, two levers control the damage. The first is keeping the pipeline full: the bubble is fixed overhead, so high concurrency (the right end of Output 24.3.1) is what makes the multi-node cost worthwhile, and a multi-node pipeline run at low traffic is the worst of both worlds, idle stages and a network tax. The second is balancing the stages: if one stage is slower than the others, it becomes the bottleneck that sets the whole pipeline's period, so you split the layers to equalize per-stage time rather than simply by layer count, accounting for the fact that the first stage also does the prefill and the last stage computes the output projection.

With the model now spread across nodes by tensor and pipeline parallelism and replicated by the router, one shared resource has been quietly growing in the background of every example: the KV cache, which stores the attention keys and values for every token of every in-flight request. At the high concurrency that makes a multi-node pipeline pay off, that cache becomes the dominant consumer of GPU memory and the true limit on how many requests a serving unit can hold. The next section takes it up directly.