Ape-X, R2D2, and SEED RL Designs

"They asked me to step the environment and also run the whole policy and also compute priorities. I am one CPU. I have opinions about this arrangement."

An Actor Carrying More Than Its Share

The previous sections built the parts. Section 20.2 separated acting from learning into the actor-learner architecture, Section 20.3 scaled experience collection across many actors, Section 20.4 turned the replay buffer into a sharded distributed service, and Section 20.5 made off-policy learning correct at scale with V-trace. Now we assemble those parts into named systems. The point of studying them together is not historical completeness; it is that each system makes one different choice on one axis, so seeing them side by side teaches the axes themselves. A practitioner who internalizes the design space can place a new system on it in a minute, and can design a fifth point that fits a workload none of the four were built for.

1. One Design Space, Four Coordinates Beginner

Every distributed-RL system in this section is a loop in which actors generate experience and a learner improves a policy from it. The systems differ in three decisions that, once you name them, become a small coordinate grid. The first axis is where policy inference runs: on each actor (the actor holds a copy of the policy and computes its own actions) or centrally (actors send raw observations and a central service computes actions in a batch). The second is how experience reaches the learner: through a replay buffer that stores and re-samples past transitions (off-policy), or as a fresh near-on-policy stream consumed once and discarded. The third is what the actor itself must compute and store, which is where recurrence bites, because an LSTM agent has hidden state that has to be carried, stored, and replayed.

These are not independent knobs you can set at random; some combinations are natural and others fight themselves. Replay demands off-policy correction because stored transitions were generated by an older policy, the exact problem V-trace solves in Section 20.5. Centralizing inference only pays off when the policy is large enough that batching many observations on one accelerator beats running it many times on small actor devices, the tradeoff this section measures. Recurrence forces a choice about hidden state that neither feed-forward replay nor on-policy streaming had to make. Table 20.6.1 places the four systems on the grid, and the rest of the section walks each coordinate and explains why its neighbors are where they are.

2. IMPALA and Ape-X: Many Actors, One Learner Intermediate

IMPALA is the template the others vary. It runs many actors that each hold a recent copy of the policy, step their environments, and ship trajectory fragments to a single central learner. Because actors run slightly stale policy copies while the learner keeps updating, the stream that arrives is not quite on-policy, and IMPALA corrects that lag with the V-trace targets developed in Section 20.5. The contribution is the shape: a centralized learner with high accelerator utilization fed by a swarm of cheap actors, with a principled correction that keeps the off-by-a-few-updates staleness from biasing the gradient. Nearly every later actor-critic system at scale is a descendant of this layout.

Ape-X keeps the swarm of actors but changes the experience path. Instead of a once-consumed on-policy stream, actors write transitions into a shared prioritized replay buffer, the sharded service of Section 20.4, and the learner samples from it by priority. The decisive systems trick is that each actor computes the priority of its own transitions locally, from the temporal-difference error it already has in hand, so the buffer never has to score incoming data centrally. This decouples the scale of data generation from the scale of learning: you can run hundreds of actors filling the buffer while a single learner drains it at its own pace, and the priorities steer that single learner toward the most informative transitions. Ape-X showed that with feed-forward DQN-style agents, raw actor count alone buys large improvements, because the bottleneck was never the learner's math, it was the rate and quality of incoming experience.

3. R2D2: Recurrence Turns Replay Into a Systems Problem Advanced

R2D2 (Recurrent Replay Distributed DQN) takes the Ape-X layout and makes the agent recurrent, an LSTM, so the policy depends on history through a hidden state $h_t$. For a feed-forward agent a stored transition is self-contained: state in, action and reward out. For a recurrent agent it is not, because the action at time $t$ depended on $h_t$, which summarizes everything before $t$. Replaying a stored sequence requires knowing what the hidden state was when the data was generated, and that hidden state was produced by a policy that has since changed. This is a storage and consistency problem, not a learning-theory problem, and it is the reason R2D2 belongs in a systems chapter.

R2D2's answer has two parts that every recurrent-replay system since has borrowed. First, it stores the actor's hidden state alongside the transition sequence in the buffer, so the learner has a starting point rather than a zero. Second, because that stored state is stale (it came from an older policy and the network weights have moved), the learner does not trust it directly; it runs a burn-in, replaying the first part of the sequence purely to warm up a fresh hidden state under current weights before computing any loss. The stored state seeds the warm-up; the burn-in repairs the staleness. The cost is concrete and worth stating plainly: the replay buffer now stores sequences plus hidden-state vectors instead of single transitions, which multiplies the bytes per sample and the network traffic the buffer service of Section 20.4 must move. Recurrence does not change the actor-learner shape; it inflates what flows through it.

4. SEED RL: Move Inference Off the Actors Advanced

IMPALA, Ape-X, and R2D2 all run the policy on the actors. SEED RL questions that. If the policy is a large neural network, running it on every actor means either putting an accelerator on each actor (most of them idle, since one environment step produces one observation and a batch of one wastes the hardware) or running the big policy on a CPU (slow). Either way, as the policy grows, the actors spend more of their time on inference and the expensive accelerators are starved by tiny batches. SEED RL's move is to take inference off the actors entirely: actors become thin clients that only step environments and exchange observations and actions with a central service over fast networking. The learner machine, which already has the accelerators, batches observations from all actors into one large forward pass, computes actions, and streams them back.

The result is a different CPU/GPU split from the one in Section 20.3. Actors are pure CPU environment-steppers with no policy and no accelerator. The central accelerator runs both learning and inference, and crucially it runs inference at a batch size equal to the number of actors, which is exactly the regime accelerators are built for. The cost SEED pays is two network hops per step, an observation out and an action back, which is why it depends on fast networking to keep that latency small. The diagram in Figure 20.6.1 contrasts the two layouts, and the demonstration that follows measures when the trade pays off.

5. When Centralizing Inference Wins Intermediate

SEED's bet is only worth the two network hops when the policy is large enough that batched inference on an accelerator decisively beats per-actor inference. We can see the crossover with a pure-throughput model that needs no RL at all. Each environment step costs a fixed amount of CPU time plus one policy forward pass. In the actor-side layout, each actor runs the forward alone on a small device, so the cost grows roughly linearly with policy size and never enjoys batching. In the centralized layout, the single accelerator batches all $A$ actors' observations into one forward, so a fixed launch overhead is amortized across the batch and the per-observation cost is far lower, at the price of two network hops per step. The code below sweeps policy size and reports the aggregate step throughput of each layout.

ENV_MS = 4.0          # env.step() per actor, milliseconds (fixed)
NET_MS = 0.3          # one-way obs/action round-trip on fast networking
A = 256               # number of actors stepping in parallel

# Actor device is small: per-forward latency grows ~linearly in policy FLOPs
# with no batching benefit (each actor has a batch of one).
def actor_fwd_ms(p_mflops):
    return 0.05 * p_mflops

# The central accelerator has a fixed launch cost but is far faster per FLOP,
# and it amortizes that launch over a full batch of A observations.
def central_fwd_ms_per_obs(p_mflops, batch):
    launch = 0.20                     # kernel launch / fixed overhead, ms
    per_obs = 0.004 * p_mflops        # accelerator ~12x faster per FLOP
    return (launch + per_obs * batch) / batch

print(f"{'policy':>8} | {'actor-side':>22} | {'centralized (SEED)':>22} | winner")
print("-" * 72)
for p in [1, 5, 20, 80, 320]:        # policy size in MFLOPs per forward
    step_actor = ENV_MS + actor_fwd_ms(p)           # env + own forward, parallel
    thr_actor = A / step_actor * 1000.0             # steps/sec across all actors

    fwd_amort = central_fwd_ms_per_obs(p, A)        # batched over all A actors
    step_central = ENV_MS + 2 * NET_MS + fwd_amort  # env + 2 hops + batched fwd
    thr_central = A / step_central * 1000.0

    win = "centralized" if thr_central > thr_actor else "actor-side"
    print(f"{p:>5} MF  | {thr_actor:>12.0f} steps/s   | {thr_central:>12.0f} steps/s   | {win}")

  policy |             actor-side |     centralized (SEED) | winner
------------------------------------------------------------------------
    1 MF  |        63210 steps/s   |        55594 steps/s   | actor-side
    5 MF  |        60235 steps/s   |        55402 steps/s   | actor-side
   20 MF  |        51200 steps/s   |        54692 steps/s   | centralized
   80 MF  |        32000 steps/s   |        52024 steps/s   | centralized
  320 MF  |        12800 steps/s   |        43532 steps/s   | centralized

The numbers match the intuition exactly. With a 1-MFLOP policy there is nothing to amortize, so SEED's network hops only cost throughput and the actor-side layout wins. The crossover sits near 20 MFLOPs, and beyond it the actor-side throughput falls off a cliff while SEED's barely moves, because SEED was always running one efficient batch regardless of policy size. This is precisely the regime SEED RL was designed for: large policies, many actors, fast networking. It also explains why the earlier systems were content to run inference on the actors; with the small feed-forward networks of Ape-X-era benchmarks, the crossover had not been reached, so there was no inefficiency to fix. SEED did not contradict its predecessors; it identified the regime where their choice stopped paying off.

# pip install "ray[rllib]"
from ray.rllib.algorithms.apex_dqn import ApexDQNConfig

config = (
    ApexDQNConfig()
    .environment("CartPole-v1")
    .env_runners(num_env_runners=64)        # 64 actors filling the buffer
    .training(num_steps_sampled_before_learning_starts=50_000)
)
algo = config.build()                        # learner + sharded prioritized replay
for _ in range(100):
    algo.train()                             # actors sample, learner drains by priority

6. Reading a New System Off the Grid Intermediate

The payoff of treating these four as a design space rather than a reading list is that the grid keeps working for systems built after them. A modern large-scale RL stack for training language-model policies, for instance, almost always centralizes inference (the policy is enormous, so it sits far past the crossover), almost always streams near-on-policy data corrected like V-trace rather than replaying from a buffer (the policy changes fast and stale samples hurt), and inherits R2D2's lesson that any state the actor carries must be shipped and warmed up. Placing it is a matter of reading three coordinates, not learning a new system from zero. The grid also tells you what is missing: there are unfilled cells, such as centralized inference paired with prioritized replay for a recurrent agent, that a workload might call for, and the design space tells you exactly what that system would have to handle.

The thread continues. These actor-learner infrastructures return in Chapter 30, where many learning agents share an environment and the single-learner picture splits further, and the next section confronts the question that has hovered under all four designs: should the actors and the learner march in lockstep or run free? That synchronous-versus-asynchronous choice, foreshadowed by the staleness that V-trace and R2D2's burn-in both repair, is the subject of Section 20.7.

7. GRPO and Reinforcement Learning with Verifiable Rewards Advanced

The four systems in Table 20.6.1 were designed for game-playing agents trained against environment simulators. The policy outputs actions in a discrete or continuous control space, and the reward signal comes from the environment itself. From 2024 onward, the same actor-learner infrastructure was pressed into service for a qualitatively different workload: training large language models to reason through mathematics and code. This application exposed two costs that game-playing RL absorbed painlessly but that become critical at language-model scale. The first is the cost of the critic. PPO, the dominant algorithm for RLHF, requires a separate value network to estimate baselines for advantage computation; for a large language model this means training and storing a second model of roughly the same size, nearly doubling the memory footprint of the learner. The second is the cost of the reward model. A neural reward model trained on human preference data is itself a large network, adds inference cost per rollout step, and is prone to reward hacking: the policy learns to exploit the reward model's inaccuracies rather than developing genuine capability.

Group Relative Policy Optimization (GRPO) and Reinforcement Learning with Verifiable Rewards (RLVR) are two algorithmic changes that address these costs at the system level, and together they redraw the distributed infrastructure requirements in ways that map directly onto the design space of this section.

7.1 GRPO: Eliminating the Critic with Group Normalization

PPO computes advantages using a learned value function $V_\phi(s)$ that acts as a baseline: the advantage of action $a$ in state $s$ is roughly $r - V_\phi(s)$, the reward minus the expected return. GRPO replaces this with a purely statistical baseline: for each prompt $q$, sample a group of $G$ responses $\{o_1, \ldots, o_G\}$ from the current policy, score each with a reward $r_i$, and normalize within the group. The group-normalized advantage for response $i$ is:

where $\mu_r$ and $\sigma_r$ are the mean and standard deviation of rewards within the group. A response that scores above the group average gets a positive advantage; one that scores below gets a negative advantage. No value network is needed. The learning signal comes from the relative ranking of responses to the same prompt, not from an absolute value estimate.

The policy update applies the same clipped surrogate objective as PPO, but using these group-normalized advantages in place of the GAE estimates. For a group of $G$ outputs sampled from the old policy $\pi_{\theta_{\text{old}}}$ for prompt $q$, the GRPO loss is:

The ratio $\pi_\theta(o_i|q)/\pi_{\theta_{\text{old}}}(o_i|q)$ is the standard PPO importance weight; the $\text{clip}(\cdot, 1-\epsilon, 1+\epsilon)$ term prevents excessively large policy updates. What is absent from this expression compared to PPO is any reference to a value function. The baseline is entirely determined by the group of outputs for the same prompt, computed on the fly and discarded after the update.

7.2 RLVR: Replacing the Reward Model with a Verifier

A neural reward model trained to predict human preference scores introduces a second large network into the pipeline and, more problematically, a failure mode: as the policy improves, it begins to exploit gaps in the reward model's generalization, scoring high on the reward model while producing outputs a human would score low. This reward hacking forces practitioners into reward-model refreshes and safety filtering that compound the infrastructure complexity.

RLVR replaces the neural reward model with a deterministic verifier. For mathematics, a symbolic checker evaluates whether the final answer is correct; for code, an execution environment runs the generated program on test cases and reports pass or fail. Both return a binary reward: 1 if the answer is provably correct, 0 otherwise. Because the verifier is deterministic and correct by construction, reward hacking becomes impossible: there is no model to fool. A program either passes the test suite or it does not; a mathematical derivation either reaches the right answer or it does not. The reward signal is clean, label-free (no human annotation beyond the problem statement and the correct answer), and infinitely scalable: running more verifier calls costs only compute, not human labelers.

The combination of GRPO with RLVR produces a training loop that needs no value network, no reward model, and no human preference labels: only a policy network, a prompt dataset with verifiable answers, and a verifier. This is a significant infrastructure simplification. The learner's memory footprint halves (no critic), and the reward pipeline collapses to a function call.

7.3 Distributed Infrastructure for GRPO

GRPO uses the same actor-learner split as IMPALA and its descendants, but the workload profile is different enough that it shifts the binding constraint. In game-playing RL, rollout and learning are both computationally intensive: stepping an environment is CPU-bound, and the policy forward pass is fast for small networks. In language-model RL, generating a full response token by token from a large language model is orders of magnitude more expensive than any environment step; auto-regressive generation at thousands of tokens per response dominates the total compute. Empirically, generation accounts for roughly 80 percent of wall-clock time in a GRPO training run, not the gradient update.

This shifts the design space in two directions. First, the "actor fleet" is not a swarm of cheap CPU boxes stepping environments; it is a fleet of accelerator servers running auto-regressive inference, and the primary optimization target is inference throughput, not environment simulation rate. Inference engines built for high-throughput LLM serving, particularly vLLM and SGLang, with their continuous batching, paged KV-cache management, and speculative decoding, become the natural choice for the actor role. Some implementations colocate the inference engine with the actor process on the same accelerator pool, switching between generation mode and training mode per iteration; others place them on separate pools with weight synchronization between them.

Second, the policy weights must exist in two layouts simultaneously: the inference-optimized layout (tensor-parallel sharding tuned for autoregressive decoding, often with quantization) and the training-optimized layout (ZeRO or FSDP sharding tuned for gradient computation across data-parallel replicas). Each update step requires resharding the weights from the learner's training layout to the actor's inference layout before the next generation round, and optionally back if they are physically co-located. This weight resharding step is a collective communication problem that sits inside the actor-learner loop; it must be fast enough that it does not dominate the already-expensive generation time.

The cross-section connections are direct. The actor-learner split of Section 20.2 remains the structural template; GRPO replaces the learning algorithm inside the learner without changing the split. The throughput balance analysis of Section 20.8 applies immediately, with generation throughput replacing environment-step throughput as the sampling-side rate. The per-node efficiency techniques of Chapter 22 (mixed precision, gradient checkpointing, fused kernels) apply to the learner's gradient computation; the LLM serving infrastructure of Chapter 24 (continuous batching, paged attention, speculative decoding) applies directly to the actor's generation step. GRPO is where training and serving infrastructure converge inside one loop.

7.4 A GRPO Training Step in Code

The code below sketches one GRPO iteration on a single prompt: sample $G$ responses from the policy, score each with a verifier, compute group-normalized advantages, then compute the clipped surrogate loss. The implementation is intentionally minimal to make the group-normalization step visible; a production system would replace the toy policy and verifier with a real language model and execution environment, and would pipeline generation across an inference engine fleet.

import torch
import torch.nn.functional as F

def grpo_step(
    policy,
    old_policy,
    prompt_tokens: torch.Tensor,        # [prompt_len]
    verifier,                            # callable: response_tokens -> float reward
    G: int = 8,
    epsilon: float = 0.2,
) -> torch.Tensor:
    """
    One GRPO update step for a single prompt.

    Steps:
      1. Sample G responses from the old (frozen) policy.
      2. Score each response with a deterministic verifier (RLVR).
      3. Compute group-normalised advantages: A_hat = (r - mu) / sigma.
      4. Compute the clipped PPO surrogate loss averaged over the group.

    Returns the scalar GRPO loss for this prompt.
    No value network or reward model is used.
    """
    rewards = []
    log_probs_old = []
    log_probs_new = []

    # Step 1 and 2: sample responses and score them.
    for _ in range(G):
        with torch.no_grad():
            response_tokens, lp_old = old_policy.sample(prompt_tokens)
        reward = verifier(response_tokens)          # deterministic: 0.0 or 1.0
        lp_new = policy.log_prob(prompt_tokens, response_tokens)
        rewards.append(reward)
        log_probs_old.append(lp_old)
        log_probs_new.append(lp_new)

    rewards_t = torch.tensor(rewards, dtype=torch.float32)   # [G]
    lp_old_t  = torch.stack(log_probs_old)                   # [G]
    lp_new_t  = torch.stack(log_probs_new)                   # [G]

    # Step 3: group-normalised advantage -- no critic, no value network.
    mu    = rewards_t.mean()
    sigma = rewards_t.std().clamp(min=1e-8)        # avoid division by zero
    advantages = (rewards_t - mu) / sigma           # [G], mean 0, std 1

    # Step 4: clipped PPO surrogate loss over the group.
    ratios = torch.exp(lp_new_t - lp_old_t)        # pi_theta / pi_theta_old
    surrogate_unclipped = ratios * advantages
    surrogate_clipped   = torch.clamp(ratios, 1 - epsilon, 1 + epsilon) * advantages
    loss = -torch.mean(torch.min(surrogate_unclipped, surrogate_clipped))
    return loss

# pip install trl vllm
from trl import GRPOTrainer, GRPOConfig

def math_verifier(prompts, completions, **kwargs):
    """Return 1.0 if the last line of the completion is the correct answer."""
    correct_answers = kwargs["answer"]
    return [1.0 if c.strip().split("\n")[-1] == a else 0.0
            for c, a in zip(completions, correct_answers)]

trainer = GRPOTrainer(
    model="deepseek-ai/deepseek-math-7b-base",
    reward_funcs=math_verifier,
    args=GRPOConfig(num_generations=8, max_completion_length=512),
    train_dataset=math_dataset,
)
trainer.train()