Value Decomposition

"The team handed me one number and called it our reward. Splitting it fairly among us was apparently my problem."

A Mixing Network Doing Everyone's Bookkeeping

The previous section established centralized training with decentralized execution (CTDE) as the organizing principle for cooperative multi-agent reinforcement learning: train with a global view, then deploy policies that run on local observations alone. Section 30.5 left open the practical question of how a team that earns one shared reward decides which agent deserves credit for it. Value decomposition is the most structurally elegant answer. Instead of estimating credit after the fact, it bakes the answer into the shape of the value function: the team value is built from per-agent pieces by construction, so each agent reads off its own piece and acts on it. This section develops the two methods that defined the family, VDN and QMIX, states the property that makes them correct, shows exactly where the simpler one breaks, and runs both on a small cooperative game so the failure and the fix are visible rather than asserted.

1. One Reward, Many Decisions Beginner

Picture a team of $n$ agents acting in a Markov game (the formal setting from Section 30.2). At each step every agent $i$ receives its own observation $o_i$, picks an action $a_i$, and the environment returns a single scalar reward $r$ shared by the whole team. There is no per-agent reward to learn from; there is one number for everyone. This is the defining feature of cooperative MARL, and it creates the credit-assignment problem that Section 30.8 studies in full: when the team does well, which agent's action actually caused it, and when it does badly, who should change?

A naive fix is to give every agent the team reward and let each learn independently, the approach of Section 30.4. It fails because an agent cannot tell whether the reward moved because of its own action or because a teammate happened to act well at the same time; the learning signal is corrupted by everyone else's choices. The opposite extreme, learning one joint value $Q_{\text{tot}}(o_1,\dots,o_n,a_1,\dots,a_n)$ over the full team, is well defined but useless at execution time: it needs every agent's observation and action at once, which no single agent has when it must act. Value decomposition threads between these extremes. It learns a per-agent value $Q_i(o_i, a_i)$ that each agent can evaluate on its own, and a rule for combining the $Q_i$ into the team value $Q_{\text{tot}}$ that is used only during centralized training.

2. The Individual-Global-Max Property Intermediate

For decentralized execution to be correct, one condition must hold. Each agent will act greedily on its own value, choosing $\arg\max_{a_i} Q_i(o_i, a_i)$, with no knowledge of what the others pick. We need that collection of independent local choices to equal the action that maximizes the team value. This is the Individual-Global-Max (IGM) property, the load-bearing constraint of the whole family:

where $\mathbf{a} = (a_1,\dots,a_n)$ is the joint action and $\boldsymbol{\tau} = (\tau_1,\dots,\tau_n)$ collects each agent's local action-observation history. Read it plainly: the joint argmax over the team value factors into per-agent argmaxes. When IGM holds, an agent that maximizes its own $Q_i$ in isolation contributes its part of the globally optimal joint action, so decentralized greedy play is jointly optimal. The entire design problem becomes: choose a way to build $Q_{\text{tot}}$ from the $Q_i$ that is expressive enough to be useful yet guarantees IGM by construction. VDN and QMIX are two such constructions; Figure 30.6.1 shows the shape they share.

3. VDN: The Joint Value Is a Sum Intermediate

Value Decomposition Networks (VDN) take the simplest construction that satisfies IGM: make the team value the sum of the per-agent values,

IGM follows immediately, because a sum is maximized term by term: each $Q_i$ depends only on agent $i$'s own action, so maximizing the sum is the same as each agent maximizing its own term independently. There is nothing to coordinate at execution time; the addition does all the work. Training is a single deep-Q update on $Q_{\text{tot}}$ against the shared-reward target, and gradients flow back through the sum into each agent's network, so every agent's value is shaped by the team's experience without ever seeing a per-agent reward. This is the same regrouping trick that made data-parallel gradients exact in Section 1.1: an additive objective decomposes cleanly across the parts, and here the parts are agents rather than data shards.

The price of that simplicity is representational. A sum can only express team values that are additively separable across agents: $Q_{\text{tot}}$ must equal a sum of functions each depending on one agent alone. Many cooperative tasks are not like that. The reward for two agents reaching a rendezvous point depends on both being there at once, an interaction no sum of single-agent terms can capture. When the true team value has such couplings, VDN cannot fit it, and the per-agent argmaxes can settle on a safe but suboptimal joint action. The connection to factored value functions and coordination graphs from Section 29.5 is exact: VDN is the degenerate coordination graph with no edges, only single-agent factors, and it inherits that graph's blind spots.

4. QMIX: Monotonic Mixing Buys Expressiveness Advanced

QMIX keeps IGM while representing far richer interactions by replacing the sum with a learned mixing network. The insight is that IGM does not actually require additivity; it requires only that $Q_{\text{tot}}$ be monotonically increasing in each per-agent value. If raising any $Q_i$ never lowers $Q_{\text{tot}}$, then the joint argmax still factors into per-agent argmaxes, because pushing each $Q_i$ to its own maximum can only push $Q_{\text{tot}}$ up. Formally, QMIX enforces

where $f_{\text{mix}}$ is a small feed-forward network whose weights are constrained to be non-negative, which is exactly what makes every partial derivative non-negative and therefore preserves IGM. The mixer also takes the global state $s$, but only through a separate side channel (a hypernetwork that generates the mixer's weights), so $s$ shapes how the per-agent values combine without breaking the monotonic dependence on the $Q_i$ themselves. VDN is the special case where the mixer is a fixed sum; QMIX lets the combination be a learned, state-dependent, monotone nonlinearity, which can represent any team value whose optimum is consistent with each agent's local ranking, a strictly larger class than the additively separable one.

QMIX is not the end of the line. Its monotonicity constraint still rules out genuinely non-monotonic payoffs, where raising one agent's local value should sometimes lower the team value (think of a task that punishes one agent for acting boldly unless a partner commits too). QTRAN reformulates the problem as a constrained optimization that targets the full IGM class without the monotonicity restriction, and QPLEX uses a duplex dueling decomposition to represent the complete IGM function class while keeping efficient greedy action selection. Both pay in training complexity for the extra expressiveness, and in practice QMIX remains the workhorse because its constraint is cheap and its failures are rare on real cooperative benchmarks. Table 30.6.1 summarizes the family.

5. Watching VDN Fail and QMIX Succeed Intermediate

The representational gap between a sum and a monotone mixer is easy to state and easy to doubt, so we make it concrete on a two-agent cooperative matrix game. Each agent has three actions; the team earns a shared reward $R[a,b]$ that both agents see, but at execution each agent must pick its action from its own per-agent value alone. The code below fits per-agent values $Q_1$ and $Q_2$ under two mixers, the additive VDN sum and a small monotone two-layer QMIX network with non-negative weights, then checks whether the decentralized argmax (each agent maximizing its own $Q_i$) recovers the true team optimum, which is exactly the IGM test.

import numpy as np

# A two-agent cooperative one-shot matrix game. Each agent has 3 actions.
# The TEAM reward R[a, b] is shared; both agents see only their own action
# at execution time, so each must learn a per-agent utility Q_i(action).
# Decentralized greedy play picks argmax_a Q_1(a) and argmax_b Q_2(b)
# independently; we want that joint choice to be the team-optimal (a*, b*).

np.set_printoptions(precision=2, suppress=True)

def fit_per_agent_Q(R, mixer, iters=4000, Q1=None, Q2=None):
    """Fit per-agent utilities Q1[a], Q2[b] so the reconstructed joint value
    mixer(Q1[a], Q2[b]) matches the shared team reward R[a, b]."""
    nA, nB = R.shape
    if Q1 is None: Q1 = np.zeros(nA)
    if Q2 is None: Q2 = np.zeros(nB)
    for _ in range(iters):                     # coordinate descent on the utilities
        for a in range(nA):
            Q1[a] -= 0.05 * np.mean([mixer.grad1(Q1[a], Q2[b]) *
                       (mixer.value(Q1[a], Q2[b]) - R[a, b]) for b in range(nB)])
        for b in range(nB):
            Q2[b] -= 0.05 * np.mean([mixer.grad2(Q1[a], Q2[b]) *
                       (mixer.value(Q1[a], Q2[b]) - R[a, b]) for a in range(nA)])
    return Q1, Q2

class VDN:                                     # joint Q = Q1 + Q2 (additive)
    name = "VDN  (additive: Q1 + Q2)"
    def value(self, q1, q2): return q1 + q2
    def grad1(self, q1, q2): return 1.0
    def grad2(self, q1, q2): return 1.0

def softplus(x): return np.log1p(np.exp(-np.abs(x))) + np.maximum(x, 0.0)
def sigmoid(x):  return 1.0 / (1.0 + np.exp(-x))

class QMIX:
    # A small TWO-LAYER monotone mixer. The hidden unit softplus(w1*Q1+w2*Q2+b1)
    # uses non-negative weights w1,w2 >= 0, and the output v*hidden+b2 uses v >= 0.
    # Non-negative weights times an increasing softplus keep d Q_tot / d Q_i >= 0,
    # so IGM is preserved, while the nonlinearity captures non-additive coupling.
    name = "QMIX (monotone 2-layer mixer, non-neg weights)"
    def __init__(self):
        self.lw1, self.lw2 = 0.0, 0.0          # log-weights -> w = softplus(.) >= 0
        self.b1, self.lv, self.b2 = 0.0, 0.0, 0.0
    def _w(self): return softplus(self.lw1), softplus(self.lw2), softplus(self.lv)
    def value(self, q1, q2):
        w1, w2, v = self._w()
        return v * softplus(w1*q1 + w2*q2 + self.b1) + self.b2
    def grad1(self, q1, q2):                    # d value / d q1 (>= 0 -> monotone)
        w1, w2, v = self._w(); return v * sigmoid(w1*q1 + w2*q2 + self.b1) * w1
    def grad2(self, q1, q2):
        w1, w2, v = self._w(); return v * sigmoid(w1*q1 + w2*q2 + self.b1) * w2
    def fit_mixer(self, Q1, Q2, R):             # one gradient step on the mixer
        nA, nB = R.shape; w1, w2, v = self._w(); g = np.zeros(5)
        for a in range(nA):
            for b in range(nB):
                z = w1*Q1[a] + w2*Q2[b] + self.b1
                h, s = softplus(z), sigmoid(z)
                e = v*h + self.b2 - R[a, b]
                g[0] += e * v*s*Q1[a] * sigmoid(self.lw1)
                g[1] += e * v*s*Q2[b] * sigmoid(self.lw2)
                g[2] += e * v*s
                g[3] += e * h * sigmoid(self.lv)
                g[4] += e
        g /= (nA*nB)
        self.lw1 -= 0.05*g[0]; self.lw2 -= 0.05*g[1]; self.b1 -= 0.05*g[2]
        self.lv  -= 0.05*g[3]; self.b2  -= 0.05*g[4]

def report(R, mixer):
    if isinstance(mixer, QMIX):                 # alternate utility and mixer updates
        Q1 = np.zeros(R.shape[0]); Q2 = np.zeros(R.shape[1])
        for _ in range(3000):
            Q1, Q2 = fit_per_agent_Q(R, mixer, iters=8, Q1=Q1, Q2=Q2)
            mixer.fit_mixer(Q1, Q2, R)
    else:
        Q1, Q2 = fit_per_agent_Q(R, mixer)
    a_hat, b_hat = int(np.argmax(Q1)), int(np.argmax(Q2))   # decentralized argmax
    a_opt, b_opt = np.unravel_index(np.argmax(R), R.shape)   # true joint optimum
    ok = (a_hat, b_hat) == (a_opt, b_opt)
    print(f"  {mixer.name}")
    print(f"    learned Q1 = {Q1},  Q2 = {Q2}")
    print(f"    decentralized argmax (a,b) = ({a_hat},{b_hat}),  team optimum = ({a_opt},{b_opt})")
    print(f"    IGM holds (greedy = joint optimum)? {'YES' if ok else 'NO'}")
    return ok

# Game 1: an additively separable team reward, R[a,b] = a + b. A sum represents
# it exactly, so VDN suffices and QMIX matches it.
R_add = np.array([[0.0, 1.0, 2.0],
                  [1.0, 2.0, 3.0],
                  [2.0, 3.0, 4.0]])
# Game 2: a coordination payoff with a tempting "safe" corner. Action 0 for both
# agents is broadly decent (reward 6), but the true team optimum is the sharp
# (2,2) peak worth 8. IGM still holds, so a monotone mixer can represent it, but
# additive VDN fits squared error across the whole table: the broad action-0
# payoffs inflate Q_i[0], so VDN's argmax settles on the SAFE corner and misses
# the peak. The monotone QMIX mixer reshapes the surface and recovers (2,2).
R_coord = np.array([[6.0, 4.0, 2.0],
                    [4.0, 3.0, 1.0],
                    [2.0, 1.0, 8.0]])

print("=== Game 1: additive team reward ===")
print(R_add)
report(R_add, VDN()); report(R_add, QMIX())

print("\n=== Game 2: non-additive coordination reward ===")
print(R_coord)
g2_vdn  = report(R_coord, VDN())
g2_qmix = report(R_coord, QMIX())

print("\nSummary:")
print("  additive game: both VDN and QMIX recover the joint optimum.")
print(f"  coordination game: VDN succeeds? {g2_vdn} ; QMIX succeeds? {g2_qmix}")

=== Game 1: additive team reward ===
[[0. 1. 2.]
 [1. 2. 3.]
 [2. 3. 4.]]
  VDN  (additive: Q1 + Q2)
    learned Q1 = [0.03 1.03 2.03],  Q2 = [-0.03  0.97  1.97]
    decentralized argmax (a,b) = (2,2),  team optimum = (2,2)
    IGM holds (greedy = joint optimum)? YES
  QMIX (monotone 2-layer mixer, non-neg weights)
    learned Q1 = [0.33 2.16 3.9 ],  Q2 = [0.3  2.14 3.88]
    decentralized argmax (a,b) = (2,2),  team optimum = (2,2)
    IGM holds (greedy = joint optimum)? YES

=== Game 2: non-additive coordination reward ===
[[6. 4. 2.]
 [4. 3. 1.]
 [2. 1. 8.]]
  VDN  (additive: Q1 + Q2)
    learned Q1 = [2.32 0.99 1.99],  Q2 = [2.23 0.9  1.9 ]
    decentralized argmax (a,b) = (0,0),  team optimum = (2,2)
    IGM holds (greedy = joint optimum)? NO
  QMIX (monotone 2-layer mixer, non-neg weights)
    learned Q1 = [-6.02 -6.66  1.33],  Q2 = [-5.97 -6.62  1.43]
    decentralized argmax (a,b) = (2,2),  team optimum = (2,2)
    IGM holds (greedy = joint optimum)? YES

Summary:
  additive game: both VDN and QMIX recover the joint optimum.
  coordination game: VDN succeeds? False ; QMIX succeeds? True

The result is the representational claim made tangible. When the team reward is additively separable, a sum is all you need and VDN is correct. When the reward couples the agents, here a tempting safe corner competing with a sharp coordinated peak, the additive form cannot fit the surface, and the agents acting independently choose the safe corner instead of coordinating on the peak. The monotone mixer, with no more information, reshapes how the per-agent values combine and restores the optimal joint choice. The same two lines that broke VDN are the case QMIX was built to handle.

# Train QMIX on a StarCraft II micromanagement map; swap --config to vdn or iql.
python src/main.py --config=qmix --env-config=sc2 with env_args.map_name=3s5z

6. Where Decomposition Sits, and Where It Stops Advanced

Value decomposition is a value-based family: it learns per-agent action values and acts greedily, which makes it a natural fit for discrete-action cooperative tasks and a close cousin of the deep-Q methods it generalizes. Its limits are the limits of monotonic factorization. Tasks with genuinely non-monotonic coordination, where one agent's locally better action is only globally better if a partner also changes, fall outside QMIX's representable class, which is what motivates QTRAN, QPLEX, and the relaxations beyond them. For continuous actions, or for problems where an explicit policy is preferable to argmax over a value, the multi-agent policy-gradient methods of Section 30.7 take over, using a centralized critic in place of a mixing network. The two families share the CTDE skeleton and differ in what the centralized object computes: a factored value here, a shared critic there.

With value decomposition in hand, the cooperative-MARL toolkit has a value-based answer to credit assignment by structure. The next section turns to the policy-gradient branch of CTDE, where a centralized critic guides decentralized actors and continuous actions come back into reach, beginning in Section 30.7. The credit-assignment question that value decomposition answered structurally returns in its own right, studied directly, in Section 30.8.