Topology-Aware Placement

"They gave me rank 5 and put me three switches away from rank 4. Every step, I wait for a packet that has to cross half the building. My neighbors on the same board finished an hour ago."

A GPU Idling on a Communication Barrier

By now you can name the collectives that distributed training runs on, and in Section 4.8 you saw the libraries (NCCL, MPI, Gloo) that implement them. Those libraries are fast, but they cannot change one fact: a collective is only as quick as the slowest link its ranks must cross. A GPU does not care, when it issues an all-reduce, whether its partner is a centimeter away on the same circuit board or a hundred meters away through three network switches. The cost model of Chapter 3 does care, because the bandwidth term changes by more than an order of magnitude between those two cases. So the question this section answers is not "which collective?" but "given the collective, where should its ranks live, and how should the operation be staged across the interconnect's levels?"

Recall the alpha-beta model from Section 3.8: sending $b$ bytes over a link costs roughly $\alpha + b/\beta$, a fixed latency $\alpha$ plus a bandwidth term, where $\beta$ is the link's bandwidth. A ring all-reduce over $n$ ranks runs $2(n-1)$ steps, each moving a $1/n$ share of the buffer, so its total time is

where $B$ is the buffer size in bytes. The structure of the formula is fixed, but $\alpha$ and $\beta$ are properties of the link, and the link is chosen by placement. Put the ring on NVLink and $\beta$ is hundreds of gigabytes per second; scatter it across nodes and $\beta$ collapses to the network rate. That single substitution is the whole subject of this section.

1. The Same Collective, Two Very Different Costs Beginner

The cleanest way to feel topology is to price one fixed operation, an eight-way all-reduce of a one-gigabyte gradient buffer, under two placements. In the first, the scheduler puts all eight ranks on a single node, so the ring lives entirely on NVLink. In the second, the eight ranks land on eight separate nodes, so every hop of the ring crosses the network. Nothing about the algorithm changes; only $\beta$ changes, and with it the elapsed time. The demo below builds both cases from the alpha-beta formula and reports the gap.

# Topology-aware placement: where a rank sits decides what its collective costs.
# Alpha-beta model: moving b bytes over a link costs alpha + b/bandwidth.
# A ring all-reduce over n ranks runs 2*(n-1) steps, each moving a (1/n)-sized
# chunk; the ring is gated by the SLOWEST link it must span.

bytes_per_elem = 4
P = 256 * 1024 * 1024           # 256M-parameter gradient buffer
B = P * bytes_per_elem          # total bytes

# Link model: (one-way latency seconds, bandwidth bytes/sec).
NVLINK  = (1.0e-6, 300e9)       # intra-node NVSwitch fabric, ~300 GB/s
NETWORK = (5.0e-6, 12.5e9)      # cross-node 100 Gb/s InfiniBand, ~12.5 GB/s

def ring_allreduce_time(n, link, B):
    alpha, bw = link
    steps = 2 * (n - 1)
    chunk = B / n                            # each step moves one (1/n) chunk
    return steps * (alpha + chunk / bw)

# Same 8-way all-reduce, placed well vs badly.
good8 = ring_allreduce_time(8, NVLINK, B)    # all 8 ranks share one node
bad8  = ring_allreduce_time(8, NETWORK, B)   # 8 ranks scattered across 8 nodes

print(f"8 ranks on one node, all NVLink   : {good8*1e3:8.2f} ms")
print(f"8 ranks scattered, all on network : {bad8*1e3:8.2f} ms")
print(f"speedup from good placement       : {bad8/good8:8.1f}x")

Running this with C:\Python314\python.exe produced the numbers below, taken verbatim from the run.

8 ranks on one node, all NVLink   :     6.28 ms
8 ranks scattered, all on network : 150.39 ms
speedup from good placement       :     24.0x

Twenty-four times is not a rounding effect; it is the bandwidth ratio between NVLink and the network, amplified slightly by the higher per-step latency of the network. For a training step that issues this all-reduce once per iteration, the difference between $6$ milliseconds and $150$ milliseconds is the difference between communication hiding behind compute and communication dominating the step. The scheduler made that decision when it chose where rank 5 would live, long before the first gradient was computed.

2. Hierarchical Collectives: Reduce Local, Exchange Global Intermediate

Most real jobs cannot fit on one node, so some traffic must cross the network. The trick is to cross it as little as possible. A hierarchical collective exploits the two-level structure of the interconnect: it first reduces within each node over the fast link, then exchanges only the reduced result across nodes over the slow link, then broadcasts the final answer back down inside each node. Instead of all $n$ ranks meeting on the network, only one leader per node does, so the slow stage runs over the number of nodes rather than the number of GPUs. Figure 4.9.1 sketches the three stages.

The demo below extends Code 4.9.1 with a hierarchical variant and prices a thirty-two-GPU all-reduce two ways: a flat ring whose ranks are scattered across thirty-two nodes, and a hierarchical collective on four eight-GPU nodes. Both move the same buffer; they differ only in how they use the topology.

def hierarchical_time(nodes, gpus_per_node, B):
    # Stage 1: ring-reduce inside every node over NVLink (nodes run in parallel).
    t_intra = ring_allreduce_time(gpus_per_node, NVLINK, B)
    # Stage 2: one leader per node all-reduces across nodes over the network.
    t_inter = ring_allreduce_time(nodes, NETWORK, B)
    # Stage 3: broadcast the reduced result back down each node over NVLink.
    a_nv, bw_nv = NVLINK
    t_bcast = a_nv + B / bw_nv
    return t_intra + t_inter + t_bcast

nodes, gpus_per_node = 4, 8
world = nodes * gpus_per_node                       # 32 GPUs
flat32 = ring_allreduce_time(world, NETWORK, B)     # all 32 ranks on the network
hier32 = hierarchical_time(nodes, gpus_per_node, B) # NVLink + a thin network layer

print(f"flat ring, scattered (bad)  : {flat32*1e3:8.2f} ms")
print(f"hierarchical NVLink+network : {hier32*1e3:8.2f} ms")
print(f"speedup from hierarchy      : {flat32/hier32:8.2f}x")

flat ring, scattered (bad)  :   166.74 ms
hierarchical NVLink+network :   138.74 ms
speedup from hierarchy      :     1.20x

The honest reading of Output 4.9.2 is worth dwelling on. Hierarchy buys a large win when an operation is latency-bound, because it slashes the number of network hops from $2(n-1)$ down to $2(\text{nodes}-1)$; it buys a smaller win when an operation is bandwidth-bound, because a reduced copy of the buffer must still traverse the slow link no matter how the ranks are grouped. The dramatic twenty-four-fold gap in Output 4.9.1 came from keeping the traffic entirely off the network; once any data must cross it, the network's bandwidth sets a floor. Both effects are real, and a good placement chases both: keep what can stay local local, and minimize what must go global.

3. Mapping Parallelism Onto the Topology Intermediate

Large models are trained with several forms of parallelism at once, and each form runs a different collective with a different traffic volume. Tensor parallelism, which splits a single layer across devices, issues an all-reduce on every forward and backward pass and is the heaviest communicator. Data parallelism issues one gradient all-reduce per step. Pipeline parallelism sends only activations between adjacent stages, the lightest of the three. The placement rule follows directly: assign the heaviest collective to the fastest link. Keep tensor-parallel groups inside a node on NVLink; let data-parallel and pipeline groups span nodes over the network, where their lighter traffic can tolerate the slower link.

Table 4.9.1 states this mapping as a checklist. It is the same logic that the three-dimensional parallelism strategies of Chapter 16 formalize, where tensor, pipeline, and data dimensions are deliberately laid out so that the innermost (most communication-intensive) dimension maps to the innermost (fastest) level of the interconnect.

The fourth row, expert parallelism, is the awkward case: its all-to-all (introduced in Section 4.6) moves tokens between every pair of expert shards and is both large and bursty, so it wants the fast link but rarely fits inside one node. That tension is exactly what Chapter 16 and the expert-parallel material that follows it must navigate, and it is one reason rail-optimized network designs (Section 5 below) exist.

4. From Placement to the Scheduler Advanced

Topology awareness is not only a property a job sets for itself; it is a property the cluster scheduler must respect. A job that needs its eight tensor-parallel ranks on one node is useless if the scheduler hands it eight ranks on eight different nodes, and a synchronous training job that runs one straggler-bound collective per step cannot tolerate having even one rank placed far away. This is why schedulers for AI clusters offer gang scheduling (all ranks of a job start together or not at all) and topology-aware placement (ranks of a group are co-located on the same node or rack). Those mechanisms are the cluster-side counterpart of everything in this section, and they get their full treatment in Chapter 33.

The contract between job and scheduler is a small piece of information: the job declares its communication groups and how tightly each must be packed, and the scheduler honors that when it assigns ranks to hardware. A tensor-parallel group declares "keep these eight together on one node"; a data-parallel group declares "these can span nodes." When the declaration is missing or ignored, you get the scenic-route all-reduce from the fun note above. When it is honored, the cost model of Chapter 3 becomes predictive rather than aspirational.

import torch.distributed as dist

dist.init_process_group("nccl")                 # NCCL probes NVLink/PCIe/network here
# Declare the tensor-parallel group: these ranks should share a node.
tp_group = dist.new_group(ranks=[0, 1, 2, 3, 4, 5, 6, 7])
# A data-parallel group whose members may span nodes.
dp_group = dist.new_group(ranks=[0, 8, 16, 24])

# The same all_reduce call; NCCL routes it hierarchically over the real topology.
dist.all_reduce(local_grad, group=tp_group)     # stays on NVLink if ranks 0-7 co-locate

5. The Interconnect Is Getting Smarter Advanced

Placement matters more every year, because the gap between the fastest and slowest links keeps widening and the fabrics keep gaining structure that a placement strategy can exploit. Three current directions are worth naming, all of which a topology-aware job must reason about.

We now have the placement rule (heaviest collective on the fastest link), the mechanism that exploits topology automatically (hierarchical collectives), and the cluster-side contract that makes placement happen (gang and topology-aware scheduling). One large lever remains unpulled. Even a perfectly placed all-reduce still costs time, and that time is wasted if the GPUs sit idle waiting for it. The next section shows how to hide the collective entirely by overlapping it with the backward pass that produces the gradients, so that communication and computation run at once. That technique, and the gradient bucketing that enables it, is the subject of Section 4.10.