Frameworks and Practice

"I have been a hand-rolled actor loop, a Ray RLlib config, and a six-thousand-GPU RLHF pipeline. The architecture never changed; only the size of the broadcast did."

An Actor-Learner That Has Worn Many Frameworks

The previous section, Section 20.8, ended on a diagnosis: a distributed-RL system is a pipeline of two interleaved workloads, sampling and learning, and its throughput is set by whichever stage is slower. You do not, in practice, write that pipeline from raw sockets. You reach for a framework that has already solved process-group setup, replay sharding, weight broadcast, and fault recovery, and that lets you spend your attention on the one stage that binds. The skill this section builds is reading a framework as an implementation of the actor-learner architecture, so that when you open Ray RLlib or OpenRLHF you already know what every component is and which dial changes which number.

1. Ray RLlib: The Architecture as a Library Beginner

Ray RLlib is the most direct mapping of this chapter's architecture onto a framework, because RLlib is built on Ray, and Ray's distributed primitives are exactly the actor model the architecture needs. A Ray actor is a stateful worker process addressable across the cluster; an RLlib rollout worker is one such actor that holds an environment and a policy copy and returns batches of experience. The learner is another Ray actor (or a group of them, for multi-GPU learning) that owns the optimizer. Ray's object store moves experience and weights between them without you touching a socket, and Ray's placement groups pin actors to the right machines. The actor-learner diagram you have been reading all chapter is, almost line for line, RLlib's internal object graph. We meet Ray as a cluster-computing substrate in its own right in Chapter 33, and Ray Serve as an inference fabric in Chapter 23; here we use only its actor primitive, the unit that makes RLlib's rollout-worker fan-out a few lines instead of a networking project.

# pip install "ray[rllib]"
from ray.rllib.algorithms.appo import APPOConfig

config = (
    APPOConfig()                                  # async actor-learner (IMPALA-style)
    .environment("CartPole-v1")
    .env_runners(num_env_runners=8)               # 8 vectorized actors = the fan-out
    .training(
        train_batch_size=2000,                    # learner minibatch
        vtrace=True,                              # off-policy correction for policy lag
    )
)
algo = config.build()                             # wires actors + learner + broadcast
for _ in range(50):
    result = algo.train()                         # one round: sample -> learn -> sync
    print(result["env_runners"]["episode_return_mean"])

2. The Other Stacks, and Where Each One Wins Intermediate

RLlib is general; several other stacks specialize, and the specialization is always a bet on which stage of Figure 20.9.1 dominates. Stable Baselines3 is the single-machine baseline: clean reference implementations of PPO, SAC, and DQN that run in one process. It is the right starting point and the correct answer whenever no resource ceiling from Chapter 1 actually binds, and you should reach for it before any distributed stack so that you have a correctness yardstick. Acme, from DeepMind, is a research framework that separates actors, learners, and replay into composable modules, and it pairs with Reverb, a dedicated distributed replay-buffer service that implements the prioritized, version-tagged storage of Section 20.4 as a standalone server other processes call into. When the replay buffer is your bottleneck, Reverb is the specialist.

When the sampling stage dominates, two stacks attack throughput directly. SampleFactory pushes a single machine to extreme rollout rates (tens to hundreds of thousands of environment steps per second) by keeping actors, a policy-inference worker, and the learner in tightly coupled processes with shared-memory queues, the asynchronous design of Section 20.7 taken to its single-node limit. EnvPool attacks the same bottleneck from the environment side: it runs many environment instances in a C++ thread pool behind a batched, vectorized interface, so a Python learner sees one fast vectorized environment instead of thousands of slow ones. EnvPool composes with the others; it is the sampling-stage accelerator you bolt onto whatever learner you already have. Table 20.9.1 places these stacks on the architecture.

3. The LLM-RL Stacks: Inference and Training, Fused Advanced

Reinforcement learning from human feedback, and its newer sibling, reinforcement learning for reasoning, are the reason distributed RL has become a frontier concern again. The architecture is unchanged: a policy generates experience, a learner improves it, weights broadcast back. What changes is that the policy is a large language model, so the sampling stage is itself a distributed-inference problem, the very subject of Chapter 24. An RLHF actor does not step a cheap simulator; it runs autoregressive generation through a sharded model, often served by an inference engine such as vLLM. The learner, meanwhile, runs a sharded training step using the foundation-model machinery of Section 19.8 in Chapter 19. The hard engineering is the fusion: keeping a fast inference deployment and a heavy training deployment on the same cluster, and broadcasting updated weights from the trainer into the inference engine quickly enough that policy lag stays bounded.

The stacks that solve this fusion are the current center of gravity. TRL (Hugging Face) is the accessible entry point, with PPO, DPO, and GRPO trainers that run on a single multi-GPU node and integrate with the wider Hugging Face ecosystem. OpenRLHF and veRL are the scale-out stacks: both place the inference (rollout) engines and the training engines as separate Ray-scheduled deployments and engineer the weight-broadcast path between them, which is precisely the broadcast arrow of Figure 20.9.1 turned into a multi-hundred-GPU operation. NeMo-Aligner (NVIDIA) brings the same fusion to the Megatron-Core training stack for the largest models. In all four, the off-policy correction of Section 20.5 reappears: because generation and training run asynchronously, the samples the learner consumes were produced by a slightly older policy, and the importance-weighting that pays for that lag is the same idea you met for IMPALA, now applied to token sequences.

4. A Decision Guide: Three Forks Intermediate

Choosing a stack reduces to three forks, taken in order. Each one is a question this chapter taught you to answer with numbers, not taste.

Fork one: single-machine or distributed? This is the Chapter 1 question applied to RL. If your environment is cheap, your model fits on one accelerator, and one machine's rollout rate keeps the learner fed, you are done: use Stable Baselines3 and do not pay the communication and failure taxes of distribution. You distribute only when a specific ceiling binds, when the environment is too slow (reach for EnvPool or SampleFactory), when the model does not fit (reach for a sharded learner), or when one machine cannot generate experience fast enough (fan out actors with RLlib). Measure the imbalance from Section 20.8 before you scale; an unbalanced single machine does not become balanced by adding machines.

Fork two: on-policy PPO or off-policy replay? On-policy methods (PPO) require that the data used for an update came from the current policy, which forces tight synchronization and caps the replay ratio near one; they are stable and are the default for RLHF. Off-policy methods (DQN, SAC, Ape-X, IMPALA) reuse a replay buffer, allowing high replay ratios and large actor fleets, but they pay for the resulting policy lag with the off-policy correction of Section 20.5. The fork is a throughput-versus-stability trade: choose off-policy when sampling is expensive and you must reuse experience, on-policy when stability matters more than sample reuse.

Fork three: classic RL or LLM-RL? If your policy is a small network stepping an environment, the classic stacks (RLlib, Acme, SampleFactory) apply directly. If your policy is a language model, the sampling stage becomes distributed inference and you need an LLM-RL stack (TRL for one node, OpenRLHF or veRL for many, NeMo-Aligner for the largest), because only those engineer the inference-training fusion and the weight-broadcast path that classic stacks never had to. Misreading this fork is the most expensive mistake: running an LLM policy through a classic stack's rollout worker ignores the distributed-inference problem that is most of the cost.

5. The Whole Chapter, Running Intermediate

To close, we assemble the entire chapter into one minimal stack and run it: vectorized actors that roll out with a local copy of the weights and a version tag, a ring replay buffer that stores version-tagged transitions, a learner that samples minibatches and bumps the policy version, and a weight broadcast that pushes fresh parameters back to the actors. The point is not the toy task (a linear contextual bandit, so the math stays out of the way) but the instrumentation: the same code reports the three numbers this chapter is about, the sampling throughput in samples per second, the replay ratio (how many times the average sample is reused by the learner), and the mean policy lag (how many learner updates stale the consumed experience is). Code 20.9.2 is the structure of Figure 20.9.1 in pure Python.

import time, collections
import numpy as np
rng = np.random.default_rng(0)

D, N_ACTIONS = 4, 2
W_ENV = rng.standard_normal((N_ACTIONS, D))               # hidden environment weights

def reward(states, actions):                              # contextual-bandit reward
    return (np.einsum("bd,bd->b", states, W_ENV[actions])
            + 0.05 * rng.standard_normal(states.shape[0]))

class Policy:                                             # linear Q(s,a) = 
    def __init__(self): self.theta = np.zeros((N_ACTIONS, D)); self.version = 0
    def act(self, s, eps=0.0):
        greedy = np.argmax(s @ self.theta.T, axis=1)
        mask = rng.random(s.shape[0]) < eps
        return np.where(mask, rng.integers(0, N_ACTIONS, s.shape[0]), greedy)

class VectorActor:                                       # holds a LOCAL weight copy + version
    def __init__(self, batch): self.batch = batch; self.theta = np.zeros((N_ACTIONS, D)); self.ver = 0
    def sync(self, p): self.theta = p.theta.copy(); self.ver = p.version   # weight broadcast
    def rollout(self):
        s = rng.standard_normal((self.batch, D))
        greedy = np.argmax(s @ self.theta.T, axis=1)
        mask = rng.random(self.batch) < 0.1
        a = np.where(mask, rng.integers(0, N_ACTIONS, self.batch), greedy)
        return s, a, reward(s, a), np.full(self.batch, self.ver)          # tag with version

class Replay:                                            # version-tagged ring buffer
    def __init__(self, cap): self.buf = collections.deque(maxlen=cap)
    def add(self, s, a, r, v):
        for it in zip(s, a, r, v): self.buf.append(it)
    def sample(self, n):
        idx = rng.integers(0, len(self.buf), size=n); items = [self.buf[i] for i in idx]
        return (np.stack([i[0] for i in items]), np.array([i[1] for i in items]),
                np.array([i[2] for i in items]), np.array([i[3] for i in items]))

class Learner:                                           # gradient step, bumps version
    def __init__(self, p, lr=0.05): self.p = p; self.lr = lr
    def step(self, s, a, r):
        for act in range(N_ACTIONS):
            m = a == act
            if m.any():
                grad = 2.0 * (s[m].T @ (s[m] @ self.p.theta[act] - r[m])) / m.sum()
                self.p.theta[act] -= self.lr * grad
        self.p.version += 1

# ---- assemble the stack and run it
N_ACTORS, ACTOR_BATCH, LEARNER_BATCH, REUSE, SYNC_EVERY, ROUNDS = 4, 256, 512, 4, 2, 200
policy = Policy(); learner = Learner(policy); replay = Replay(50_000)
actors = [VectorActor(ACTOR_BATCH) for _ in range(N_ACTORS)]
for ac in actors: ac.sync(policy)

samples = updates = lag_sum = lag_n = 0
t0 = time.perf_counter()
for rnd in range(ROUNDS):
    for ac in actors:                                    # sampling stage
        s, a, r, v = ac.rollout(); replay.add(s, a, r, v); samples += len(r)
    if len(replay.buf) >= LEARNER_BATCH:                 # learning stage
        for _ in range(REUSE):
            s, a, r, v = replay.sample(LEARNER_BATCH)
            lag_sum += int((policy.version - v).sum()); lag_n += len(v)   # policy lag
            learner.step(s, a, r); updates += 1
    if rnd % SYNC_EVERY == 0:                            # weight broadcast stage
        for ac in actors: ac.sync(policy)
elapsed = time.perf_counter() - t0

ts = rng.standard_normal((10_000, D))
v_greedy = reward(ts, policy.act(ts)).mean()
v_rand = reward(ts, rng.integers(0, N_ACTIONS, 10_000)).mean()
print(f"actors                : {N_ACTORS} x batch {ACTOR_BATCH}")
print(f"samples collected     : {samples:,}")
print(f"learner updates       : {updates:,}")
print(f"wall-clock (s)        : {elapsed:.3f}")
print(f"samples/sec           : {samples / elapsed:,.0f}")
print(f"replay ratio (reuse)  : {updates * LEARNER_BATCH / samples:.2f}")
print(f"mean policy lag       : {lag_sum / max(lag_n, 1):.2f} learner updates")
print(f"value greedy vs random: {v_greedy:+.3f} vs {v_rand:+.3f}")

actors                : 4 x batch 256
samples collected     : 204,800
learner updates       : 800
wall-clock (s)        : 0.894
samples/sec           : 229,148
replay ratio (reuse)  : 2.00
mean policy lag       : 87.46 learner updates
value greedy vs random: +0.542 vs -0.007

The three balance numbers in Output 20.9.2 are exactly the quantities Section 20.8 told you to watch, and three knobs in Code 20.9.2 move them; together they are the chapter in miniature. Raising REUSE lifts the replay ratio and the sample efficiency but also raises the policy lag, because old samples are reused more before fresh weights arrive. Lowering SYNC_EVERY broadcasts weights more often and cuts the lag, at the cost of more synchronization, the sync-versus-async trade of Section 20.7. Adding actors raises samples per second until the learner cannot keep up, at which point the imbalance of Section 20.8 returns and more actors stop helping. You can now turn every dial in this chapter and watch the three numbers move.

6. Chapter Summary Beginner

This chapter treated reinforcement learning as a distributed-systems problem and built its infrastructure from the ground up. The through-line is a single architecture refracted through eight concerns. RL is two interleaved workloads, sampling (actors generating experience) and learning (a learner improving the policy), and the actor-learner architecture of Section 20.2 separates them so each can scale on its own. Experience collection (Section 20.3) fans out across many actors, and the replay buffer (Section 20.4) decouples the rate at which experience is produced from the rate at which it is consumed, storing version-tagged transitions for reuse. Because actors run on slightly stale weights, the learner consumes off-policy data, and the off-policy correction of Section 20.5 (V-trace and its relatives) reweights that data so the lag does not bias the update. The landmark designs of Section 20.6 (Ape-X, R2D2, SEED RL) are points in this design space; the synchronous-versus-asynchronous choice of Section 20.7 trades determinism against throughput; and the scaling analysis of Section 20.8 showed that the whole system runs at the speed of its slower stage, so balancing the sampling and learning pipeline is the central engineering task. This section closed the loop by mapping every framework onto that one architecture and running the full stack to measure the three numbers, throughput, replay ratio, and policy lag, that the balance comes down to.

7. Exercises Intermediate