Communication Libraries: NCCL, MPI, and Gloo

"You spent four sections deciding which collective to call. I am the one who actually has to move the bytes, pick the ring, and find the wire. A little credit would be nice."

A Communication Backend That Does the Real Work

Across the previous five sections we treated collectives as mathematical objects: an all-reduce sums one vector per worker and gives every worker the result; reduce-scatter and all-gather split that work for sharded training; all-to-all routes tokens between experts. None of those sections said how a vector of four billion bytes actually crosses from one GPU to another, in what order, over which link, and overlapped with which part of the backward pass. That "how" is the job of a communication library. The library is the layer that sits below your training framework and above the hardware, and the quality of its implementation is often the difference between a job that scales to a thousand GPUs and one that stalls at sixty four. Section 4.7 closed the catalogue of primitives; this section names the engines that run them.

1. Why a Separate Library Layer Exists at All Beginner

A reasonable question is why the framework does not just send the bytes itself. The answer is that a correct, fast collective is a substantial piece of systems engineering that no training framework wants to own. To move a gradient with an all-reduce, something must decide whether to use a ring or a tree algorithm (a choice that depends on message size and worker count, as Section 4.4 showed), detect that two GPUs share an NVLink bridge while a third is a network hop away, fall back from RDMA to TCP when InfiniBand is absent, register pinned memory buffers, and schedule the transfer so it overlaps with the parts of the backward pass that have already finished. Getting all of that right, and portable, and fast, is a multi-year effort. The communication libraries exist so that effort is written once and reused by every framework.

This is the same separation of concerns that lets Chapter 15 describe data-parallel training without ever mentioning a wire protocol: the framework expresses intent ("synchronize these gradients"), and the library expresses mechanism ("ring all-reduce over NVLink within the node, tree all-reduce over InfiniBand across nodes"). The cost models of Chapter 3 are how you predict what that mechanism will cost; the library is what actually pays the cost. Keeping the two layers distinct is what lets you change interconnects without rewriting your model.

2. The Three Libraries, and What Each Is For Beginner

Three libraries cover nearly every distributed AI job, and they divide the space cleanly by where the data lives and what hardware connects it. NCCL (the NVIDIA Collective Communications Library) is purpose-built for collectives among NVIDIA GPUs. It is topology-aware down to the level of individual NVLink bridges and PCIe switches, drives RDMA over InfiniBand and RoCE for cross-node transfers, and is the default and recommended backend for essentially all GPU training. When this book says "the all-reduce" in a deep-learning context, NCCL is almost always the code running it.

MPI (the Message Passing Interface) is the standard of classical High Performance Computing, decades old and extremely mature. It runs collectives across CPUs and, through CUDA-aware builds, across GPUs as well. On a supercomputer or a national-lab cluster that already speaks MPI and is provisioned with an MPI-aware job launcher, using the MPI backend lets an AI job inherit the cluster's tuned transport stack. Outside that setting, MPI is less common for pure GPU deep learning, because NCCL usually matches or beats it on NVIDIA hardware and needs no separate installation.

Gloo is Meta's portable collective library. It targets CPUs first, supports a useful subset of collectives on GPUs, and asks nothing of the environment beyond ordinary TCP, which makes it the dependable fallback. You reach for Gloo when there is no GPU, when you are debugging a distributed program on a laptop, or when a collective you need runs on CPU tensors (for example, gathering Python metadata across workers). It is rarely the fast path, and that is fine, because it is the path you take when correctness and portability matter more than peak bandwidth. Table 4.8.1 lays the three side by side.

3. Selecting a Backend in torch.distributed Intermediate

The reason you rarely think about these libraries by name is that torch.distributed presents them as interchangeable backends behind one collective API. You pass a backend string to init_process_group, and from that point every all_reduce, all_gather, or all_to_all call is routed to the library you named. Swapping NCCL for Gloo is a one-word change with no edit to the collective calls themselves, which is precisely the portability the library layer buys you. The widely used rule of thumb is short: NCCL for GPU collectives, Gloo for CPU work and debugging, MPI when you are on an HPC cluster that already runs it.

The program below probes which backends the installed PyTorch was compiled with, applies that rule of thumb to pick one, and then shows what a sum all-reduce computes over a small group. On a multi-GPU box the same code would initialize NCCL and run the all-reduce over the network; on the single CPU host used to capture the output, it reports the available backends and simulates the collective so the semantics are visible without a second process.

import torch
import torch.distributed as dist

# 1. Which backends did THIS PyTorch build compile in? (a real probe)
for b in ["nccl", "mpi", "gloo"]:
    available = getattr(dist, "is_" + b + "_available")()
    print(f"  {b:5s}: {'available' if available else 'not available'}")

# 2. Rule of thumb: NCCL for GPU collectives, else Gloo for CPU/debugging.
chosen = ("nccl" if torch.cuda.is_available() and dist.is_nccl_available()
          else "gloo" if dist.is_gloo_available() else "none")
print("device available   :", "cuda" if torch.cuda.is_available() else "cpu-only")
print("selected backend   :", chosen)

# 3. What a SUM all-reduce computes, shown for a 3-worker group.
#    Single-host simulation: each "worker" holds its own gradient vector;
#    the collective replaces every copy with the element-wise sum.
workers = [torch.tensor([1., 2., 3.]),
           torch.tensor([0., 1., 0.]),
           torch.tensor([4., 0., 2.])]
result = torch.stack(workers).sum(dim=0)   # what all_reduce(SUM) yields on every worker
print("per-worker inputs  :", [w.tolist() for w in workers])
print("all_reduce(SUM)    :", result.tolist(), "(held by every worker afterward)")
print("mean (divide by 3) :", (result / 3).tolist())

  nccl : not available
  mpi  : not available
  gloo : available
device available   : cuda
selected backend   : gloo
per-worker inputs  : [[1.0, 2.0, 3.0], [0.0, 1.0, 0.0], [4.0, 0.0, 2.0]]
all_reduce(SUM)    : [5.0, 3.0, 5.0] (held by every worker afterward)
mean (divide by 3) : [1.6666666269302368, 1.0, 1.6666666269302368]

Two details in that output are worth dwelling on. First, NCCL availability is a property of the build, not just the hardware: NCCL is a Linux library, so a Windows PyTorch can see a GPU yet report NCCL unavailable, which is why the rule of thumb checks is_nccl_available() and not only torch.cuda.is_available(). Second, the all-reduce result is identical no matter which backend produces it; the backend changes the speed and the wire, never the arithmetic. That invariance is what makes the backend a tuning knob rather than a correctness concern.

# Run with: torchrun --nproc_per_node=8 train.py
import torch.distributed as dist

dist.init_process_group(backend="nccl")   # GPU training: topology-aware, RDMA-capable
# dist.init_process_group(backend="gloo") # swap one word to debug on CPU
# dist.init_process_group(backend="mpi")  # or to run under an HPC launcher

# Every collective below is now routed to the chosen library, unchanged:
dist.all_reduce(grad, op=dist.ReduceOp.SUM)

4. What the Library Handles for You Intermediate

It pays to name the four things the library does on your behalf, because each one connects to material elsewhere in this chapter and book. The first is algorithm selection. For a small gradient the library may pick a tree-based reduction to keep latency low; for a large one it switches to the bandwidth-optimal ring all-reduce that Section 4.4 explained, and modern NCCL can even split a single collective across both NVLink and the network simultaneously. You expressed "all-reduce"; the library chose "ring or tree" using the very cost trade-offs of Chapter 3.

The second is topology awareness. The library inspects the machine to learn which GPUs share an NVLink bridge, which sit behind the same PCIe switch, and which are reachable only across the network, then builds its communication pattern to keep as much traffic as possible on the fastest links. This is so consequential that Section 4.9 is devoted to placing your workers so the library has a good topology to exploit in the first place. The third is transport: the library opens RDMA queues over InfiniBand or RoCE when they exist, uses shared memory for same-host transfers, and falls back to TCP otherwise, all without a line of socket code from you. The fourth is overlap, launching the transfer on a separate stream so it proceeds while computation continues, the mechanism that Section 4.10 turns into gradient bucketing during the backward pass.

5. Where the Libraries Sit Under the Frameworks Advanced

It helps to see the whole stack at once, because the libraries of this section are a thin and well-defined layer with a great deal resting on top of them. Above torch.distributed sit the training frameworks: DistributedDataParallel and FSDP in PyTorch, then DeepSpeed and Megatron-LM, then orchestration systems like Ray. Every one of those frameworks ultimately expresses its parallelism as collective calls, which means every one of them runs on NCCL, MPI, or Gloo underneath. When Chapter 16 shards a model with FSDP and issues reduce-scatter and all-gather, those collectives are NCCL calls; the chapter can reason about parallelism strategy precisely because this layer reliably turns each collective into bytes on the fastest available wire.

This is why the chapter places the libraries in Part I rather than burying them inside a training chapter. They are shared infrastructure. The same NCCL that synchronizes gradients in data-parallel training (Chapter 15) also moves shards in FSDP (Chapter 16) and routes tokens in expert parallelism. Learning the backend layer once means every later parallel method reads as "which collective, over which backend, on which topology," a question you can now answer. With the engines named, the next question is how to arrange the workers so those engines find a friendly topology, which is exactly where Section 4.9 goes.