Independent Learners

"I trained beautifully against the partner I imagined. Unfortunately the real one kept learning too, and my replay buffer is now a museum of policies nobody plays anymore."

An Independent Learner, Surprised Again

The previous sections built the formal stage: a Markov game (Section 30.2) in which several agents act at once, and the cooperative, competitive, and mixed reward structures (Section 30.3) that decide whether those agents are allies, rivals, or both. The honest question now is how to actually learn a policy in that setting. Before reaching for any machinery designed for many agents, it is worth asking what happens if we ignore the multi-agent structure entirely and simply deploy the single-agent reinforcement-learning algorithms we already trust. The answer is more interesting than it sounds, and it sets the baseline that every more elaborate method in this chapter must beat.

Independent learning is exactly that move. Each agent runs a standard single-agent algorithm, Q-learning or PPO, against the stream of observations and rewards it happens to receive, and it never represents the existence of the others. The two canonical instances have names: Independent Q-Learning (IQL), where each agent maintains its own action-value function, and Independent PPO (IPPO), where each agent runs its own clipped policy-gradient update. The rest of this section is about why this naive idea scales so beautifully, why it is theoretically broken, and why, despite being broken, it keeps winning benchmark tables.

1. The Embarrassingly Parallel Ideal Beginner

The reason independent learning deserves the first word is that it is the cheapest thing to distribute in this entire chapter. Recall the data-parallel identity from Section 1.1: distributing training was attractive precisely because the combine step (an all-reduce of gradients) was exact and bounded. Independent learning goes one step further and removes the combine step altogether. There is no gradient to all-reduce across agents, no parameter server to consult, no shared replay to synchronize. Each agent's learner is a self-contained process that reads its own observations, computes its own loss, and updates its own parameters. The only thing the agents share is the environment they happen to inhabit together.

Concretely, agent $i$ keeps an action-value estimate $Q_i(s, a_i)$ over its own action set and applies the ordinary single-agent Q-learning update,

using only the local reward $r_i$ and its own next-state value. Nothing in this update references another agent's action, policy, or value function; the joint action of the others is folded silently into the transition from $s$ to $s'$ and into the reward $r_i$. That is the whole trick, and it is what makes the scheme trivially distributed: the update for agent $i$ has the identical shape as a single-agent update, so any existing single-agent RL stack, including the distributed actor-learner infrastructure of Chapter 20, runs it unchanged, once per agent.

Figure 30.4.1 makes the structural bargain visible. The wiring is identical to a single-agent loop, which is exactly why it distributes for free: replicate the orange box $n$ times, give each its own slice of the world's feedback, and you have an $n$-agent learner with zero inter-agent communication during training. The blue boxes inside the dashed boundary are the price of that simplicity, and the next sections are about what they cost.

2. Why the Foundation Is Cracked: Non-Stationarity Intermediate

Single-agent Q-learning converges to the optimal value function because it is fitting a fixed object: the environment's transition and reward functions do not change while you learn. Independent learning quietly violates that assumption. From agent $i$'s perspective the effective transition dynamics are obtained by marginalizing out the other agents' actions under their current policies,

where $a_{-i}$ denotes the joint action of everyone except $i$. The true Markov game $P$ is fixed, but $\pi_j$ is being updated by every other learner, so $P_i^{\text{eff}}$ drifts over the course of training. Agent $i$ is therefore solving a non-stationary problem with a tool that assumes stationarity, and the comforting convergence theorem of single-agent Q-learning simply does not apply. We treat the full theory of this drift, and the methods built to tame it, in Section 30.9; here we only need the consequence: learning can oscillate, chase its own tail, or settle into a poor joint policy, and which of these happens can depend on something as innocent as the random initialization.

The damage is worst for off-policy methods that reuse old experience. A replay buffer (the workhorse of deep Q-learning) stores transitions $(s, a_i, r_i, s')$ for later reuse, on the assumption that an old transition still describes the current environment. Under independent learning that assumption is false: a transition recorded ten thousand steps ago was generated when the other agents played a policy they have since abandoned, so its reward and next state reflect a world that no longer exists. The buffer becomes an inconsistent mixture of experience drawn from many vanished opponent policies, and replaying it teaches the agent to be good against ghosts. This is why on-policy methods such as IPPO, which discard experience after each update, tend to behave far better in practice than off-policy IQL with a long replay buffer.

3. Watching Non-Stationarity Decide the Outcome Intermediate

The cleanest way to feel the effect is the smallest possible cooperative game. Two agents each choose one of two actions and receive a shared reward: the joint action $(1,1)$ pays the most ($5$), the joint action $(0,0)$ is a safe-but-worse equilibrium ($3$), and any miscoordination pays nothing. This is a one-state coordination game (a Stag-Hunt-flavored payoff), and it is cooperative in the sense of Section 30.3: both agents want the same outcome. The code below runs two independent tabular Q-learners on it, with no model of each other, and asks a single question: starting from $200$ different random initializations, how often do the two learners actually converge to the optimal joint policy rather than the mediocre one?

import numpy as np

# A 2x2 cooperative coordination game (a "Stag Hunt"-like joint-reward payoff).
# Both agents pick action 0 or 1 and get a SHARED reward for the joint action.
# The best joint action is (1,1)=5; (0,0)=3 is a safe but worse equilibrium.
PAYOFF = np.array([[3.0, 0.0],   # this agent plays 0: partner 0 -> 3, partner 1 -> 0
                   [0.0, 5.0]])  # this agent plays 1: partner 0 -> 0, partner 1 -> 5

def run(seed, episodes=4000, alpha=0.1, eps=0.1):
    rng = np.random.default_rng(seed)
    Q = rng.uniform(0, 1, size=(2, 2))   # Q[agent, action]; random init per seed
    for _ in range(episodes):
        a = [int(np.argmax(Q[i])) if rng.random() > eps else rng.integers(2)
             for i in range(2)]          # epsilon-greedy, each agent independent
        r = PAYOFF[a[0], a[1]]           # shared reward from the joint action
        for i in range(2):               # independent Q-update, no partner model
            Q[i, a[i]] += alpha * (r - Q[i, a[i]])
    greedy = (int(np.argmax(Q[0])), int(np.argmax(Q[1])))
    return greedy, PAYOFF[greedy[0], greedy[1]]

outcomes = {}
for seed in range(200):
    joint, val = run(seed)
    outcomes[joint] = outcomes.get(joint, 0) + 1

print("joint policy reached (over 200 random initializations):")
for joint in sorted(outcomes, key=lambda k: -outcomes[k]):
    label = "optimal (1,1)=5" if joint == (1, 1) else (
            "safe   (0,0)=3" if joint == (0, 0) else "miscoordinated =0")
    print(f"  {joint} -> {outcomes[joint]:3d} runs   [{label}]")
opt = outcomes.get((1, 1), 0)
print(f"reached the optimal joint policy: {opt}/200 = {opt/200:.0%} of seeds")

joint policy reached (over 200 random initializations):
  (0, 0) -> 104 runs   [safe   (0,0)=3]
  (1, 1) ->  96 runs   [optimal (1,1)=5]
reached the optimal joint policy: 96/200 = 48% of seeds

The lesson of Output 30.4.1 is stark precisely because the game is so easy: a single state, two actions, fully shared reward, perfect information. A coordinating pair could trivially agree on $(1,1)$. Yet two independent learners reach it only $48\%$ of the time, with the outcome decided by initialization, because each agent is climbing its own value estimate while the partner shifts the ground underneath. When agent $1$ tentatively favors action $1$ but agent $0$ has not yet, the miscoordination reward of $0$ punishes action $1$, reinforcing the retreat to the safe $(0,0)$ basin. Whether the pair escapes that basin is a matter of which agent commits first, which is set by the initial Q-values. This is the non-stationarity penalty of Section 30.9 in miniature, and it is the entire argument for the centralized training of Section 30.5.

4. Why the Cracked Baseline Keeps Winning Advanced

Given Output 30.4.1, you might expect independent learning to be a strawman that real methods crush. The empirical record refuses to cooperate with that story. On many standard cooperative benchmarks, Independent PPO matches or beats considerably more elaborate algorithms that were specifically designed to handle the multi-agent structure. There are two reasons, and they pull in the same direction. First, IPPO is on-policy, so it sidesteps the worst of the stale-replay problem from Section 2; each agent always learns from experience generated under the current opponent policies, which keeps its effective environment locally close to stationary. Second, PPO's trust-region clipping keeps each policy update small, which slows the rate at which any one agent moves the others' environment, giving the population time to track a shared trajectory rather than chasing wild swings.

The practical upshot is a strong default: before deploying a method with a centralized critic, a mixing network, or a learned communication channel, run IPPO and make the complicated method prove it is worth the added coupling. Independent learning is the right answer more often than its broken theory suggests, especially when rewards are dense, the number of agents is large enough that centralized training is expensive, and the task does not demand tight, brittle coordination. It is the wrong answer when the optimal policy requires agents to commit to a joint action no individual would choose alone, exactly the basin-escape failure of Output 30.4.1, where the moving-target penalty is not a nuisance but the whole difficulty.

# pip install "ray[rllib]"
from ray.rllib.algorithms.ppo import PPOConfig

# Independent learners: every agent gets its OWN policy (separate weights + optimizer).
def per_agent(agent_id, *args, **kwargs):
    return agent_id                       # one policy id per agent -> IPPO

config = (
    PPOConfig()
    .environment("my_marl_env")           # any MultiAgentEnv (PettingZoo, custom, ...)
    .multi_agent(
        policies={"agent_0", "agent_1", "agent_2"},   # independent policies
        policy_mapping_fn=per_agent,                   # no sharing, no central critic
    )
    .env_runners(num_env_runners=8)       # 8 parallel samplers; agents never sync grads
)
algo = config.build()
for _ in range(100):
    algo.train()                          # each policy runs its own PPO update

5. When Independence Suffices and When It Does Not Intermediate

The decision rule that emerges is concrete enough to apply before writing any code. Reach for independent learning first when the agents' rewards are dense and well aligned with progress, when the population is large enough that centralized training would be expensive or fragile, when on-policy updates are acceptable so stale replay never accumulates, and when the optimal behavior does not hinge on a brittle joint commitment that no single agent would risk alone. In those regimes the moving-target penalty stays small, the embarrassingly parallel structure of Section 1 pays off directly, and the simplicity is a feature rather than a liability.

Move to the centralized training with decentralized execution of Section 30.5 when the opposite holds: when rewards are sparse or shared in a way that demands credit assignment across agents (Section 30.8), when the optimal policy requires the basin-escaping coordination that Output 30.4.1 showed independence fumbles, or when adversarial non-stationarity makes each learner's target swing too violently to track. The recurring narrative of distributed RL infrastructure (Chapter 20) returns here transformed: the same actor-learner fan-out that scaled single-agent RL now scales a population of agents, and independent learning is the configuration where that fan-out needs no synchronization at all. The next section keeps the decentralized execution but reintroduces a carefully bounded amount of centralized information at training time, buying back exactly the coordination that this section showed independence cannot guarantee. That story, centralized training with decentralized execution, begins in Section 30.5, and it will reappear at fleet scale in the multi-robot case study of Chapter 39.