Ape-X, R2D2, and SEED RL Designs

"They asked me to step the environment and also run the whole policy and also compute priorities. I am one CPU. I have opinions about this arrangement."

An Actor Carrying More Than Its Share

The previous sections built the parts. Section 20.2 separated acting from learning into the actor-learner architecture, Section 20.3 scaled experience collection across many actors, Section 20.4 turned the replay buffer into a sharded distributed service, and Section 20.5 made off-policy learning correct at scale with V-trace. Now we assemble those parts into named systems. The point of studying them together is not historical completeness; it is that each system makes one different choice on one axis, so seeing them side by side teaches the axes themselves. A practitioner who internalizes the design space can place a new system on it in a minute, and can design a fifth point that fits a workload none of the four were built for.

1. One Design Space, Four Coordinates Beginner

Every distributed-RL system in this section is a loop in which actors generate experience and a learner improves a policy from it. The systems differ in three decisions that, once you name them, become a small coordinate grid. The first axis is where policy inference runs: on each actor (the actor holds a copy of the policy and computes its own actions) or centrally (actors send raw observations and a central service computes actions in a batch). The second is how experience reaches the learner: through a replay buffer that stores and re-samples past transitions (off-policy), or as a fresh near-on-policy stream consumed once and discarded. The third is what the actor itself must compute and store, which is where recurrence bites, because an LSTM agent has hidden state that has to be carried, stored, and replayed.

These are not independent knobs you can set at random; some combinations are natural and others fight themselves. Replay demands off-policy correction because stored transitions were generated by an older policy, the exact problem V-trace solves in Section 20.5. Centralizing inference only pays off when the policy is large enough that batching many observations on one accelerator beats running it many times on small actor devices, the tradeoff this section measures. Recurrence forces a choice about hidden state that neither feed-forward replay nor on-policy streaming had to make. Table 20.6.1 places the four systems on the grid, and the rest of the section walks each coordinate and explains why its neighbors are where they are.

2. IMPALA and Ape-X: Many Actors, One Learner Intermediate

IMPALA is the template the others vary. It runs many actors that each hold a recent copy of the policy, step their environments, and ship trajectory fragments to a single central learner. Because actors run slightly stale policy copies while the learner keeps updating, the stream that arrives is not quite on-policy, and IMPALA corrects that lag with the V-trace targets developed in Section 20.5. The contribution is the shape: a centralized learner with high accelerator utilization fed by a swarm of cheap actors, with a principled correction that keeps the off-by-a-few-updates staleness from biasing the gradient. Nearly every later actor-critic system at scale is a descendant of this layout.

Ape-X keeps the swarm of actors but changes the experience path. Instead of a once-consumed on-policy stream, actors write transitions into a shared prioritized replay buffer, the sharded service of Section 20.4, and the learner samples from it by priority. The decisive systems trick is that each actor computes the priority of its own transitions locally, from the temporal-difference error it already has in hand, so the buffer never has to score incoming data centrally. This decouples the scale of data generation from the scale of learning: you can run hundreds of actors filling the buffer while a single learner drains it at its own pace, and the priorities steer that single learner toward the most informative transitions. Ape-X showed that with feed-forward DQN-style agents, raw actor count alone buys large improvements, because the bottleneck was never the learner's math, it was the rate and quality of incoming experience.

3. R2D2: Recurrence Turns Replay Into a Systems Problem Advanced

R2D2 (Recurrent Replay Distributed DQN) takes the Ape-X layout and makes the agent recurrent, an LSTM, so the policy depends on history through a hidden state $h_t$. For a feed-forward agent a stored transition is self-contained: state in, action and reward out. For a recurrent agent it is not, because the action at time $t$ depended on $h_t$, which summarizes everything before $t$. Replaying a stored sequence requires knowing what the hidden state was when the data was generated, and that hidden state was produced by a policy that has since changed. This is a storage and consistency problem, not a learning-theory problem, and it is the reason R2D2 belongs in a systems chapter.

R2D2's answer has two parts that every recurrent-replay system since has borrowed. First, it stores the actor's hidden state alongside the transition sequence in the buffer, so the learner has a starting point rather than a zero. Second, because that stored state is stale (it came from an older policy and the network weights have moved), the learner does not trust it directly; it runs a burn-in, replaying the first part of the sequence purely to warm up a fresh hidden state under current weights before computing any loss. The stored state seeds the warm-up; the burn-in repairs the staleness. The cost is concrete and worth stating plainly: the replay buffer now stores sequences plus hidden-state vectors instead of single transitions, which multiplies the bytes per sample and the network traffic the buffer service of Section 20.4 must move. Recurrence does not change the actor-learner shape; it inflates what flows through it.

4. SEED RL: Move Inference Off the Actors Advanced

IMPALA, Ape-X, and R2D2 all run the policy on the actors. SEED RL questions that. If the policy is a large neural network, running it on every actor means either putting an accelerator on each actor (most of them idle, since one environment step produces one observation and a batch of one wastes the hardware) or running the big policy on a CPU (slow). Either way, as the policy grows, the actors spend more of their time on inference and the expensive accelerators are starved by tiny batches. SEED RL's move is to take inference off the actors entirely: actors become thin clients that only step environments and exchange observations and actions with a central service over fast networking. The learner machine, which already has the accelerators, batches observations from all actors into one large forward pass, computes actions, and streams them back.

The result is a different CPU/GPU split from the one in Section 20.3. Actors are pure CPU environment-steppers with no policy and no accelerator. The central accelerator runs both learning and inference, and crucially it runs inference at a batch size equal to the number of actors, which is exactly the regime accelerators are built for. The cost SEED pays is two network hops per step, an observation out and an action back, which is why it depends on fast networking to keep that latency small. The diagram in Figure 20.6.1 contrasts the two layouts, and the demonstration that follows measures when the trade pays off.

5. When Centralizing Inference Wins Intermediate

SEED's bet is only worth the two network hops when the policy is large enough that batched inference on an accelerator decisively beats per-actor inference. We can see the crossover with a pure-throughput model that needs no RL at all. Each environment step costs a fixed amount of CPU time plus one policy forward pass. In the actor-side layout, each actor runs the forward alone on a small device, so the cost grows roughly linearly with policy size and never enjoys batching. In the centralized layout, the single accelerator batches all $A$ actors' observations into one forward, so a fixed launch overhead is amortized across the batch and the per-observation cost is far lower, at the price of two network hops per step. The code below sweeps policy size and reports the aggregate step throughput of each layout.

ENV_MS = 4.0          # env.step() per actor, milliseconds (fixed)
NET_MS = 0.3          # one-way obs/action round-trip on fast networking
A = 256               # number of actors stepping in parallel

# Actor device is small: per-forward latency grows ~linearly in policy FLOPs
# with no batching benefit (each actor has a batch of one).
def actor_fwd_ms(p_mflops):
    return 0.05 * p_mflops

# The central accelerator has a fixed launch cost but is far faster per FLOP,
# and it amortizes that launch over a full batch of A observations.
def central_fwd_ms_per_obs(p_mflops, batch):
    launch = 0.20                     # kernel launch / fixed overhead, ms
    per_obs = 0.004 * p_mflops        # accelerator ~12x faster per FLOP
    return (launch + per_obs * batch) / batch

print(f"{'policy':>8} | {'actor-side':>22} | {'centralized (SEED)':>22} | winner")
print("-" * 72)
for p in [1, 5, 20, 80, 320]:        # policy size in MFLOPs per forward
    step_actor = ENV_MS + actor_fwd_ms(p)           # env + own forward, parallel
    thr_actor = A / step_actor * 1000.0             # steps/sec across all actors

    fwd_amort = central_fwd_ms_per_obs(p, A)        # batched over all A actors
    step_central = ENV_MS + 2 * NET_MS + fwd_amort  # env + 2 hops + batched fwd
    thr_central = A / step_central * 1000.0

    win = "centralized" if thr_central > thr_actor else "actor-side"
    print(f"{p:>5} MF  | {thr_actor:>12.0f} steps/s   | {thr_central:>12.0f} steps/s   | {win}")

  policy |             actor-side |     centralized (SEED) | winner
------------------------------------------------------------------------
    1 MF  |        63210 steps/s   |        55594 steps/s   | actor-side
    5 MF  |        60235 steps/s   |        55402 steps/s   | actor-side
   20 MF  |        51200 steps/s   |        54692 steps/s   | centralized
   80 MF  |        32000 steps/s   |        52024 steps/s   | centralized
  320 MF  |        12800 steps/s   |        43532 steps/s   | centralized

The numbers match the intuition exactly. With a 1-MFLOP policy there is nothing to amortize, so SEED's network hops only cost throughput and the actor-side layout wins. The crossover sits near 20 MFLOPs, and beyond it the actor-side throughput falls off a cliff while SEED's barely moves, because SEED was always running one efficient batch regardless of policy size. This is precisely the regime SEED RL was designed for: large policies, many actors, fast networking. It also explains why the earlier systems were content to run inference on the actors; with the small feed-forward networks of Ape-X-era benchmarks, the crossover had not been reached, so there was no inefficiency to fix. SEED did not contradict its predecessors; it identified the regime where their choice stopped paying off.

# pip install "ray[rllib]"
from ray.rllib.algorithms.apex_dqn import ApexDQNConfig

config = (
    ApexDQNConfig()
    .environment("CartPole-v1")
    .env_runners(num_env_runners=64)        # 64 actors filling the buffer
    .training(num_steps_sampled_before_learning_starts=50_000)
)
algo = config.build()                        # learner + sharded prioritized replay
for _ in range(100):
    algo.train()                             # actors sample, learner drains by priority

6. Reading a New System Off the Grid Intermediate

The payoff of treating these four as a design space rather than a reading list is that the grid keeps working for systems built after them. A modern large-scale RL stack for training language-model policies, for instance, almost always centralizes inference (the policy is enormous, so it sits far past the crossover), almost always streams near-on-policy data corrected like V-trace rather than replaying from a buffer (the policy changes fast and stale samples hurt), and inherits R2D2's lesson that any state the actor carries must be shipped and warmed up. Placing it is a matter of reading three coordinates, not learning a new system from zero. The grid also tells you what is missing: there are unfilled cells, such as centralized inference paired with prioritized replay for a recurrent agent, that a workload might call for, and the design space tells you exactly what that system would have to handle.

The thread continues. These actor-learner infrastructures return in Chapter 30, where many learning agents share an environment and the single-learner picture splits further, and the next section confronts the question that has hovered under all four designs: should the actors and the learner march in lockstep or run free? That synchronous-versus-asynchronous choice, foreshadowed by the staleness that V-trace and R2D2's burn-in both repair, is the subject of Section 20.7.