Broadcast and Gather: Weight and Experience Movement

"The learner kept sending me the same weights as everyone else, then acted surprised when the experience I sent back was nothing like my neighbor's. That, it turns out, is the entire point."

An Actor Rolling Out a Slightly Stale Policy

The collectives in Section 4.3 through Section 4.6 share a quiet assumption: every participating rank plays the same role. In all-reduce every worker contributes a gradient and every worker receives the same averaged result. In all-gather every worker contributes a shard and every worker ends up with the full set. That symmetry is natural for data-parallel SGD, where the workers are interchangeable replicas of one model. Many AI systems break that symmetry on purpose. They designate one rank as special, a root, that either owns the data everyone needs or needs the data everyone owns. The two primitives that serve a root are broadcast (the root sends, everyone receives) and gather (everyone sends, the root receives), together with gather's mirror image, scatter (the root sends a different piece to each worker). This section is about that asymmetric family and the AI systems built on it.

1. Broadcast: One Model, Sent to Everyone Beginner

A broadcast takes a buffer that lives on one rank, the root, and delivers a byte-identical copy to every other rank in the group. Formally, if rank $r$ holds vector $v$ and is the root, then after $\text{broadcast}(v, \text{root}=r)$ every rank $j$ holds the same $v$. There is no arithmetic; nothing is combined or reduced. The only thing that happens is replication of one rank's data across the whole group. That sounds almost too simple to deserve a name, yet it is one of the most frequently issued collectives in real training systems, because distributed training is full of moments where one machine knows something authoritative that all the others must adopt exactly.

The first such moment happens before a single gradient is computed. When a data-parallel job starts, every worker must begin from identical model weights, otherwise the exact-gradient identity of Section 1.1 falls apart, because the workers would be computing gradients of different models. The standard recipe is to initialize the model on rank 0 and broadcast its parameters to all other ranks, guaranteeing a common starting point. The same pattern recurs whenever a new model version must be published to a fleet: one rank loads the new checkpoint and broadcasts it so that, from the next step onward, every worker runs the same model.

2. Gather and Scatter: Collecting and Distributing Distinct Data Beginner

Gather is the inverse of broadcast in direction but not in content. In a gather, every rank holds a distinct buffer and the root ends up holding the concatenation of all of them, indexed by source rank. If rank $j$ holds $v_j$, then after $\text{gather}(\text{root}=r)$ the root holds the ordered collection $(v_0, v_1, \dots, v_{K-1})$ while the non-root ranks keep only their own piece. Crucially the pieces are not combined: the root receives every worker's data individually, which is the whole point when the root needs to inspect, store, or process each contribution separately rather than only their sum.

Scatter is gather run backward. The root holds $K$ distinct pieces and sends piece $j$ to rank $j$, so that each worker receives a different slice of the root's buffer. Where broadcast sends the same data to everyone, scatter sends different data to each. The canonical use is handing out work: a coordinator that holds a batch of tasks, or a parameter server that holds a partitioned model, scatters the relevant slice to each worker so that no worker receives more than it needs. Table 4.7.1 lays the four root-centric collectives side by side so the symmetry, or rather the asymmetry, is explicit.

Reading Table 4.7.1 top to bottom is the cleanest way to see the design choice these primitives encode. If you only ever need the aggregate of what the workers computed, all-reduce is both sufficient and efficient, because it never materializes the individual contributions anywhere. The moment you need the individual contributions preserved, because the root must store each gradient, replay each trajectory, or route each piece somewhere specific, you reach for gather instead, and you accept that the root becomes a traffic and memory bottleneck that all-reduce was carefully designed to avoid.

3. First AI Use: Parameter-Server Push and Pull Intermediate

The parameter-server architecture, which Chapter 11 develops in full, is the oldest large-scale distributed-training design and it is built almost entirely from broadcast-like and gather-like movement. One set of machines, the servers, holds the authoritative model parameters. Another set, the workers, holds shards of the data. Each training step is two asymmetric movements. First a pull: every worker fetches the current parameters from the servers, which is broadcast-shaped, one authoritative copy delivered to many workers. Then a push: every worker sends its locally computed gradient back to the servers, which is gather-shaped, many distinct gradients collected at the root, where the server applies them to update the parameters.

This is a genuinely different communication pattern from the all-reduce of data-parallel SGD, and the difference is instructive. In all-reduce there is no privileged rank; the averaged gradient emerges symmetrically and every worker updates its own copy of the model. In a parameter server there is a privileged rank, the server, that owns the model and is the sole writer. That asymmetry buys flexibility, the server can apply updates asynchronously, hold a model far larger than any one worker by sharding it across many servers, and let workers join and leave, which is why parameter servers dominate the enormous sparse embedding tables of recommendation systems. It also concentrates traffic and state on the servers, the bottleneck that ring all-reduce in Section 4.4 was invented to escape for dense gradients. The choice between the two is a recurring theme, traced explicitly in the parameter-server-versus-all-reduce arc that runs from here into Chapter 15.

4. Second AI Use: Broadcasting Policies, Gathering Experience Intermediate

Distributed reinforcement learning makes the asymmetry even sharper, and it is the cleanest setting in which to see why all-reduce simply cannot do the job. The dominant architecture, the actor-learner pattern that Chapter 20 builds into a full system, separates two roles. A small number of learners hold the policy and improve it by gradient descent. A large number of actors, often hundreds, each run a copy of the policy against their own environment instance and produce experience: sequences of states, actions, and rewards. The two roles are coupled by exactly the primitives of this section. The learner broadcasts the latest policy weights to all actors, identical bytes to every actor, and then gathers the experience each actor collected, distinct batches from every actor.

The reason this cannot be an all-reduce is that the learner does not want the average of the actors' experience; it wants every actor's experience kept separate, because each trajectory is a distinct training example that the learner will replay, weight by importance, and learn from individually. Averaging rollouts would destroy the very signal reinforcement learning depends on. So the experience movement is irreducibly a gather, not a reduction. The demo below simulates one round of this loop with $K$ actors, makes the broadcast and gather concrete, and reports the distinct experience the learner collects.

import numpy as np

rng = np.random.default_rng(7)
K, d, steps_per_actor = 6, 4, 1000        # actors, policy dim, rollout length

# Learner holds the current policy weights (one vector).
policy = rng.standard_normal(d).round(3)

# --- BROADCAST: every actor receives a byte-identical copy of `policy`. ---
actor_copies = [policy.copy() for _ in range(K)]
broadcast_ok = all(np.array_equal(c, policy) for c in actor_copies)
print(f"broadcast: 1 learner -> {K} actors, weight dim = {d}")
print(f"  every actor holds identical weights : {broadcast_ok}")
print(f"  bytes sent by learner               : {K} x {policy.nbytes} = {K*policy.nbytes} B")

def rollout(weights, env_seed):
    # Each actor acts with its policy copy in its OWN environment instance,
    # so the experience it produces is distinct from every other actor's.
    r = np.random.default_rng(env_seed)
    obs = r.standard_normal((steps_per_actor, d))
    reward = obs @ weights + 0.3 * r.standard_normal(steps_per_actor)
    return {"return": float(reward.sum()), "n": steps_per_actor}

experience = [rollout(actor_copies[k], env_seed=100 + k) for k in range(K)]

# --- GATHER: the learner collects the K distinct experience batches. ---
print(f"gather: {K} actors -> 1 learner (distinct experience each)")
for k, e in enumerate(experience):
    print(f"  actor {k}: return = {e['return']:+9.2f} over {e['n']} steps")

total_steps = sum(e["n"] for e in experience)
mean_return = sum(e["return"] for e in experience) / K
print(f"  gathered batches                    : {len(experience)}")
print(f"  total experience steps              : {total_steps}")
print(f"  mean episodic return across actors  : {mean_return:+.2f}")

broadcast: 1 learner -> 6 actors, weight dim = 4
  every actor holds identical weights : True
  bytes sent by learner               : 6 x 32 = 192 B
gather: 6 actors -> 1 learner (distinct experience each)
  actor 0: return =     +6.70 over 1000 steps
  actor 1: return =    +70.80 over 1000 steps
  actor 2: return =    +51.67 over 1000 steps
  actor 3: return =    -24.22 over 1000 steps
  actor 4: return =    +31.58 over 1000 steps
  actor 5: return =     +0.17 over 1000 steps
  gathered batches                    : 6
  total experience steps              : 6000
  mean episodic return across actors  : +22.78

5. Cost, Bottlenecks, and When Symmetry Wins Advanced

The asymmetry of broadcast and gather has a price, and the cost models of Chapter 3 name it precisely. A naive broadcast in which the root sends $K$ separate copies costs the root $K$ message transmissions; under the $\alpha + \beta n$ model, sending an $n$-byte buffer to $K$ ranks one at a time costs the root roughly $K(\alpha + \beta n)$, growing linearly in the number of workers. A naive gather is worse in a subtle way: the root must receive $K$ distinct buffers, so its inbound bandwidth, not the workers', sets the floor, and that floor is $K \beta n$ no matter how the messages are scheduled, because every byte every worker produced must physically arrive at the one root. This is the concentration problem: the root is a single point that all traffic squeezes through.

Good implementations soften the broadcast cost dramatically. A tree broadcast forwards the buffer through $\log_2 K$ levels, so the data reaches all $K$ ranks in $O(\log K)$ steps rather than $O(K)$, and a pipelined or ring broadcast can overlap transmission so that the bandwidth term, not the latency term, dominates. NCCL, which Section 4.8 covers, ships exactly these optimized broadcasts. Gather is harder to accelerate because its fundamental limit is the root's inbound bandwidth, which no algorithm can route around; the only real escape is to stop using a single root. That escape is precisely what all-gather and reduce-scatter (Section 4.5) provide, spreading the role of the root across every rank so the per-rank traffic stays flat as $K$ grows. The practical rule that falls out of these costs is the subject of the example below.

6. The Library View Beginner

You almost never implement broadcast, gather, or scatter by hand. Every collective-communication library exposes them directly, with the root chosen by an argument, and handles the tree or pipeline algorithm, the buffer management, and the network transport internally. The shortcut below shows the three primitives of this section in torch.distributed, which dispatches to NCCL on GPUs and Gloo on CPUs (both covered in Section 4.8).

# Run with: torchrun --nproc_per_node=4 thisfile.py
import torch, torch.distributed as dist

dist.init_process_group("nccl")              # join the group of K workers
rank, world = dist.get_rank(), dist.get_world_size()

# BROADCAST: rank 0 owns the weights; all ranks adopt its copy in place.
weights = torch.empty(1024, device="cuda")
if rank == 0:
    weights = load_initial_weights()         # only the root has real values
dist.broadcast(weights, src=0)               # now every rank holds rank 0's weights

# GATHER: every rank sends its own experience; only the root keeps them all.
local_exp = collect_local_experience()       # this rank's distinct batch, a tensor
bucket = [torch.empty_like(local_exp) for _ in range(world)] if rank == 0 else None
dist.gather(local_exp, gather_list=bucket, dst=0)   # root's `bucket` = all K batches

# SCATTER: the root hands a different task slice to each worker.
slices = list(task_slices) if rank == 0 else None    # K distinct tensors on the root
my_slice = torch.empty(256, device="cuda")
dist.scatter(my_slice, scatter_list=slices, src=0)   # each rank receives its own slice

With the asymmetric collectives in hand, the chapter has now named every primitive the rest of the book relies on: the symmetric all-reduce, all-gather, reduce-scatter, and all-to-all of the preceding sections, and the root-centric broadcast, gather, and scatter of this one. What remains is to see how a production library actually implements them across real hardware, choosing algorithms and topologies on your behalf. That is the subject of Section 4.8, which opens the lid on NCCL, MPI, and Gloo.