Synchronous vs Asynchronous RL Systems

"I generated a beautiful trajectory under the policy from three updates ago. The learner thanked me politely and corrected most of it away."

An Actor Rolling Out Against a Lagging Policy

The previous section built out the Ape-X, R2D2, and SEED RL designs, each of which makes a specific stance on a single question: when an actor finishes a trajectory, does it wait for the learner before continuing, or not? That question looks like an implementation detail and is in fact the most consequential architectural decision in distributed RL. It governs the throughput of the whole system on one side and the statistical quality of the data feeding the learner on the other, and those two pull in opposite directions. We will treat the choice as a single axis, measure both ends of it, and then map real systems onto it.

1. The Same Tension, Now in Reinforcement Learning Intermediate

In data-parallel supervised training the synchronous-asynchronous axis is about gradients. Synchronous SGD makes every worker compute a gradient on the current parameters, averages them at a barrier with an all-reduce, and steps once; asynchronous SGD lets each worker push its gradient to a parameter server and pull whatever parameters are there, so gradients are computed against slightly stale weights. Section 10.10 framed the result as a throughput-versus-staleness trade-off: dropping the barrier raises hardware utilization but injects gradient staleness that, past a point, slows or destabilizes convergence.

Reinforcement learning has the identical axis with one extra twist. In RL the workers are actors that generate experience by acting in an environment, and the learner trains on that experience. If actors never wait, they generate trajectories under an old policy while the learner has already moved on, so the data is not merely computed against stale parameters; it is distributed differently from the policy being optimized. That is off-policy drift, and unlike gradient staleness it changes the meaning of the data, not just its freshness. Correcting it is the job of importance sampling and V-trace, built in Section 20.5. The throughput-versus-bias trade-off of RL is thus the throughput-versus-staleness trade-off of SGD with a sharper penalty for going fast.

2. Synchronous On-Policy Systems: Clean Data, Idle Machines Intermediate

A synchronous RL system runs in rounds. All actors are handed the current policy parameters $\theta_t$. Each rolls out a fixed amount of experience, the system waits at a barrier until the slowest actor is done, the learner consumes the assembled batch and produces $\theta_{t+1}$, and the round repeats. This is how synchronous PPO and A2C (the synchronous, batched form of A3C) operate. Because every sample in the batch was generated by exactly the policy being updated, the data is on-policy: the policy-gradient estimator is unbiased and the update is as stable as RL updates get. For PPO this matters doubly, because its clipped objective is derived assuming the data comes from a policy close to the current one, and a barrier guarantees that closeness.

The cost is utilization. Two distinct stalls appear in every round. First, the barrier itself: actors finish at different times because environments have variable episode lengths and machines have variable speed, so all $N$ actors wait for the slowest, the straggler problem from Section 2.7. With heavy-tailed rollout times the expected wait is governed by the maximum of $N$ samples, which grows with $N$, so adding actors gives diminishing returns. Second, the learner-update window: while the learner computes $\theta_{t+1}$, the actors hold the old policy and have nothing to do, so they sit idle. Both stalls are dead time that no amount of fast actor compute can fill.

Figure 20.7.1 makes the two stalls visible: in the top timeline the dashed gaps and the learner block are pure waste, while in the bottom timeline they vanish. That visible waste is exactly what asynchrony reclaims, and exactly what it pays for.

3. Asynchronous Systems: No Idle Machines, Stale Data Intermediate

An asynchronous RL system removes the barrier. Each actor holds a local copy of the policy, rolls out a trajectory, ships the experience to the learner (or to a replay buffer, Section 20.4), pulls the most recent published policy, and immediately starts the next rollout. The learner trains continuously on whatever arrives. A3C did this with many CPU actors each updating shared parameters; IMPALA centralized the learner on a GPU and streamed trajectories from a fleet of actors; Ape-X did it with hundreds of actors feeding one prioritized replay buffer. No actor ever waits for another, and the learner never waits for a full synchronized batch, so the system runs near the throughput its slowest single component allows rather than near the speed of its slowest actor.

The price is policy lag. By the time an actor's trajectory reaches the learner, the learner has applied several more updates, so the behavior policy $\mu$ that generated the data is several versions behind the target policy $\pi$ being optimized. A naive policy gradient computed on this data is biased, because the trajectory is sampled from $\mu$ but the gradient is meant for $\pi$. The fix is an importance-sampling correction, weighting each sample by $\pi/\mu$, which IMPALA's V-trace estimator does with truncated weights for stability (Section 20.5). The correction works only while the lag stays bounded; if actors fall arbitrarily far behind, the importance weights explode or get clipped to uselessness, and the learning signal degrades. Asynchronous systems therefore live or die by keeping policy lag small, which is why they publish fresh policies to actors frequently and cap how stale a usable sample may be.

4. Measuring Both Ends of the Axis Intermediate

The trade-off is easy to state and easy to mismeasure, so we simulate it directly. The code below models eight actors whose rollout times are heavy-tailed (most quick, a few slow stragglers) over a fixed two-second wall-clock. In the synchronous regime, every round waits for the slowest of the eight actors and then for one learner update, and all data has policy lag zero. In the asynchronous regime, the simulation is event-driven: whichever actor finishes next delivers its sample, the learner applies an update as soon as it is free, and the actor restarts under the newest policy, so each consumed sample carries a measurable policy lag. We report samples per second and average policy lag for each.

import random

random.seed(0)

N_ACTORS = 8
HORIZON = 2.0          # seconds of simulated wall-clock
LEARN_TIME = 0.010     # learner update cost, seconds
MEAN_ROLLOUT = 0.040   # mean actor rollout time, seconds

def rollout_time():
    # Heavy-tailed: most rollouts quick, a few slow stragglers.
    return MEAN_ROLLOUT * random.expovariate(1.0)

def run_sync():
    t, policy_version, samples, lag_sum = 0.0, 0, 0, 0
    while t < HORIZON:
        durations = [rollout_time() for _ in range(N_ACTORS)]
        slowest = max(durations)           # barrier ends with the straggler
        samples += N_ACTORS
        lag_sum += 0 * N_ACTORS            # every sample is on-policy: lag 0
        t += slowest + LEARN_TIME          # actors idle through both stalls
        policy_version += 1
    return samples, lag_sum, policy_version

def run_async():
    finish = [rollout_time() for _ in range(N_ACTORS)]
    started_under = [0] * N_ACTORS
    published_version, samples, lag_sum, learner_busy_until = 0, 0, 0, 0.0
    while True:
        i = min(range(N_ACTORS), key=lambda k: finish[k])   # next to finish
        t = finish[i]
        if t >= HORIZON:
            break
        lag_sum += published_version - started_under[i]      # policy lag
        samples += 1
        learner_busy_until = max(t, learner_busy_until) + LEARN_TIME
        published_version += 1
        started_under[i] = published_version                 # restart, fresh policy
        finish[i] = t + rollout_time()                       # never waits
    return samples, lag_sum, published_version

s = run_sync(); a = run_async()
print(f"{'regime':<6} {'samples':>8} {'samples/s':>10} {'updates':>8} {'avg policy lag':>15}")
print(f"{'sync':<6} {s[0]:>8} {s[0]/HORIZON:>10.1f} {s[2]:>8} {s[1]/s[0]:>15.2f}")
print(f"{'async':<6} {a[0]:>8} {a[0]/HORIZON:>10.1f} {a[2]:>8} {a[1]/a[0]:>15.2f}")
print(f"throughput ratio async/sync : {a[0]/s[0]:.2f}x")

regime  samples  samples/s  updates  avg policy lag
sync        136       68.0       17            0.00
async       393      196.5      393            6.85
throughput ratio async/sync : 2.89x

The numbers in Output 20.7.1 are the whole trade-off in one table. Asynchrony nearly tripled throughput, and it did so by trading away on-policy purity: the average sample fed to the asynchronous learner came from a policy almost seven updates behind. A synchronous PPO learner would refuse that data, or rather its clipped objective would be invalid on it; an IMPALA learner accepts it precisely because V-trace reweights it back toward on-policy (Section 20.5). The 6.85-update lag is the bias that the correction must absorb, and it is finite only because the simulated learner keeps up with the actors. Let the actors outpace the learner and that number climbs without bound, which is the failure mode Section 20.8 studies as a sampling-versus-learning throughput imbalance.

5. Where Each Regime Wins Intermediate

The choice is not a matter of taste; different problems put different weights on throughput and bias, and that determines the answer. Table 20.7.1 lays out the considerations and where each regime lands.

Synchronous PPO dominates two important settings. The first is RL from human feedback for large language models, where the policy is a multi-billion-parameter model, each update is expensive, and stability and reproducibility matter more than squeezing out the last unit of actor throughput; PPO's on-policy guarantee is worth the barrier. The second is many continuous-control tasks (robotics, locomotion), where sample efficiency and stable updates beat raw sample count. Asynchronous designs dominate the opposite regime: massive-throughput settings such as large-scale Atari or simulated environments where samples are cheap, the policy is small enough to update very fast, and the bottleneck is simply getting enough experience through the system. There IMPALA-style streaming with V-trace, or Ape-X-style prioritized replay, turns hundreds of actors into a firehose the learner can drink from.

6. Scaling Synchronous PPO With Data-Parallel Learners Advanced

The RLHF example raises an obvious worry: if synchronous PPO must stay on-policy and the policy is enormous, how does it scale at all? The answer is that the two halves of the loop scale by different means. The actor side scales out by replication, more generation workers producing more completions per round, exactly the distributed experience collection of Section 20.3. The learner side scales by data parallelism, the same all-reduce that has run through this book since Chapter 15: the large on-policy batch is split across many learner replicas, each computes the PPO gradient on its shard, and an all-reduce averages the gradients before the optimizer step. Modern large-batch synchronous PPO thus keeps the barrier and the on-policy guarantee while making both the batch and the learner arbitrarily large, which is precisely why the same recipe that trains foundation models (Chapter 19) carries over to their RL fine-tuning stage.

The result is that the synchronous-versus-asynchronous choice is not the same as small-versus-large. A synchronous system scales to thousands of accelerators by enlarging the on-policy batch and data-parallelizing the learner; its limit is the straggler-and-stall overhead this section measured, not a hard size cap. An asynchronous system scales by decoupling, letting actors and learner run at their own speeds; its limit is how much policy lag the correction can absorb. The two scaling stories sit on the two ends of one axis, and Section 20.8 turns the overheads of both into the bottleneck analysis that decides how far each can actually go.

from ray.rllib.algorithms.ppo import PPOConfig      # synchronous, on-policy
from ray.rllib.algorithms.impala import IMPALAConfig  # asynchronous, V-trace corrected

sync = (PPOConfig().environment("CartPole-v1")
        .env_runners(num_env_runners=8))             # 8 actors, barrier each round

async_ = (IMPALAConfig().environment("CartPole-v1")
          .env_runners(num_env_runners=64))          # 64 actors, never wait; V-trace on

7. Research Frontier Advanced

The synchronous-asynchronous axis is being actively renegotiated by the rise of RL for LLM reasoning, where the cost of the barrier and the cost of staleness are both being re-measured at frontier scale.

You now hold the central design axis of distributed RL: synchronous for clean on-policy data at the cost of idle machines, asynchronous for saturated machines at the cost of off-policy bias, with large-batch data-parallel learners letting the synchronous side scale and bounded-lag corrections letting the asynchronous side stay valid. What this section deliberately left as a single number, the learner keeping up with the actors, becomes the main subject next: when sampling throughput and learning throughput fall out of balance, the system bottlenecks, and which side bottlenecks decides everything. That accounting begins in Section 20.8.