Distributed Agent Orchestration

"I am an Orchestrator, holding the plan while five workers run ahead. Two have already finished, one is retrying a flaky tool, one is blocked on a lock, and one has wandered off to call an API I never approved. This is, somehow, the calm part of my job."

An Orchestrator, Holding the Plan While Five Workers Run Ahead

In the previous section we built the retrieval layer that an agent reaches into when it needs grounded facts. Now we step up a level and ask how the agent uses that retrieval, alongside code execution and external APIs, inside a multi-step reasoning loop, and how that loop becomes a distributed system once one agent is no longer enough. The progression mirrors the rest of the book. A single language model call is the scale-up baseline: capable, self-contained, and bounded by what fits in one context window and one forward pass. Once a task needs many steps, many tools, and work that can proceed in parallel, the application scales out into a coordinated set of agents and services. The discipline for doing that well is distributed agent orchestration, and it is the sixth axis of distribution from Chapter 1: distributing intelligence itself.

1. The Agent Loop: Plan, Act, Observe, Repeat Beginner

The smallest unit of agentic behavior is a loop, not a single call. A language model is given a goal and a set of available tools; it reasons about what to do next, emits an action (a tool call with arguments), receives the tool's result as an observation, and feeds that observation back into its context to decide the next action. This reason, act, observe cycle, popularized as the ReAct pattern, repeats until the model decides the goal is met and emits a final answer. The loop is what turns a static predictor into something that can gather information it did not start with, correct course when a tool returns an error, and chain several operations toward a goal no single call could reach.

Each turn of the loop is cheap to describe and expensive to run. The reasoning step is one LLM forward pass, served from the fleet of Section 40.7. The action step blocks on an external service whose latency the agent does not control: a vector search over the index of Section 40.4, a sandboxed code execution, or a third-party API. Because the model must see each observation before choosing the next action, the turns of a single agent's loop are inherently sequential, and the wall-clock time of the loop is the sum of its per-turn latencies. That sequential spine is exactly the part that orchestration tries to keep short, and the independent work hanging off it is exactly the part that orchestration tries to run in parallel.

We can put a number on how long such a loop runs. If the agent takes a random number of turns $T$ to reach its goal, and each turn that has not yet finished succeeds with probability $q$, then $T$ is geometric and the expected number of turns to completion is

where $\ell_{\text{reason}}$ is the LLM latency per turn and $\ell_{\text{act}}$ is the tool latency per turn. The formula is deliberately blunt, but it captures the operational truth: a loop that needs ten turns at half a second each is a five-second user wait, and the only ways to shrink it are to reduce the number of turns (better planning) or to reduce per-turn latency (faster tools and a faster model). When the per-turn tool work is itself splittable across machines, a third lever appears, and that lever is the subject of the rest of this section.

2. Tools as Distributed Services Beginner

Function calling, the mechanism by which a model emits a structured request to invoke a named tool with typed arguments, is the interface through which an agent reaches the rest of the system. Each tool is a service with its own latency, failure profile, and scaling story. The retrieval tool of Section 40.4 is a sharded vector index that answers in tens of milliseconds and scales by adding index replicas. A code-execution tool is a pool of sandboxed workers, each an isolated container that runs untrusted model-written code and is torn down afterward; it scales by adding sandbox capacity and is the slowest and riskiest tool in the kit. An external API, a weather service or a payments gateway, is a tool the application does not own at all, with rate limits and an availability budget set by someone else.

Because the tools are independent services, they fail and scale independently, and the orchestration layer must hold each to its own contract. A retrieval timeout should not abort a plan that could proceed with a cached result; a payments call must never be retried blindly, because a duplicate charge is worse than a failure; a code sandbox that hangs must be killed and its turn re-planned rather than waited on forever. These are the coordination and failure concerns of Chapter 2, now wearing the costume of an agent's toolbox. The agent chooses which tool to call; the engineering choice is how each tool behaves when it is slow, absent, or returns garbage.

tools = [retrieve_tool, run_code_tool, call_api_tool]   # each a typed schema

messages = [{"role": "user", "content": goal}]
while True:
    reply = model.run(messages, tools=tools)            # the Reason step
    if reply.stop_reason != "tool_use":                 # model judged it Done
        break
    for call in reply.tool_calls:                       # the Act step (may be many)
        result = dispatch(call.name, call.arguments)    # an RPC to a tool service
        messages.append(tool_result(call.id, result))   # the Observe step

3. Planner and Executor: Fanning Work Out to Role-Specialized Agents Intermediate

One agent looping over tools handles a surprising amount, but it serializes everything, and many goals decompose into independent pieces. A research request might need five sources summarized, a coding task might touch four files, a data pipeline might validate six tables. The production pattern, built in depth in Chapter 32, splits the single agent into roles: a planner (or supervisor) decomposes the goal into sub-tasks and decides their dependencies, and a set of executor agents, each possibly specialized for a role such as researcher, coder, or critic, carry out the sub-tasks. The supervisor then gathers the executors' results and either finishes or plans another round. This is a map-then-reduce over reasoning, with the supervisor playing coordinator and the executors playing workers, the same shape as the all-reduce of Chapter 1 but with intelligence rather than gradients as the payload.

Whether this fan-out actually saves time depends on how much of the plan is genuinely independent. If a fraction $p$ of the total work can run in parallel across $W$ worker agents and the remaining fraction $1-p$ is the irreducibly sequential planning spine, the speedup is bounded by Amdahl's law from Chapter 3,

The ceiling is sobering: even with infinitely many workers, a plan that is $15\%$ sequential can never run more than about $6.7$ times faster than serial execution. The lesson for agent design is the same as for parallel training: the win comes from making the parallel fraction large (decompose aggressively into independent sub-tasks) and the sequential spine short (fewer supervisor rounds), not from throwing more workers at a plan that is mostly serial. For independent sub-tasks with per-task latencies $\ell_1, \dots, \ell_n$, sequential execution costs $\sum_i \ell_i$ while parallel execution costs only $\max_i \ell_i$, so the speedup is the sum-over-max ratio, and a single slow sub-task (a hung sandbox) caps the gain exactly as a straggler caps a synchronous training step.

4. The Orchestration Engine: Durable Execution, Retries, and State Intermediate

Fan-out is easy to draw and hard to run reliably, because the moment work is spread across many agents and tool calls, partial failure becomes the normal case. A worker crashes mid-plan, a tool times out, the process serving the supervisor is preempted. A toy script loses everything; a production orchestration engine does not, because it treats the plan as durable state. Each completed sub-task is checkpointed, so a restart resumes from the last good step rather than re-running the whole plan; each tool call carries a retry policy and an idempotency guarantee, so a transient failure is retried without duplicating side effects; and the engine tracks which sub-tasks are ready (dependencies satisfied), running, or done, scheduling the ready ones onto available workers. These are precisely the fault-tolerance and coordination primitives of Chapter 2, applied to a workflow whose steps happen to be acts of reasoning.

The runtime substrate underneath is a distributed-execution framework. Chapter 33 develops Ray as exactly this kind of substrate: it places each agent and tool worker as a task or actor across the cluster, moves the results between them, and restarts the ones that die, so that the orchestration logic can speak in terms of sub-tasks and dependencies while the framework handles process placement, fault recovery, and state transport. The agentic application thus inherits the cluster-scheduling story of the rest of the book: agents are scheduled work, tool services are scaled pools, and the engine is the coordinator that keeps a fleet of cooperating reasoners behaving like one coherent system.

The demonstration below makes the fan-out payoff concrete. It models a plan of independent sub-tasks, each a tool call with its own latency, and compares the plan two ways: run sequentially through one worker, its critical path is the sum of the latencies; fanned out to a pool of workers as a supervisor would dispatch them, its critical path is the slowest single sub-task. It reports the ideal fan-out speedup against the Amdahl ceiling, so the gap between independent-task ideal and the ceiling a partly-serial plan can reach is visible.

import random

random.seed(7)

# A plan of N independent subtasks the planner discovered. Latencies vary
# because tools differ (a vector search is fast, a code sandbox is slow).
# Each subtask is a tool call (retrieval, code exec, or an external API),
# and `secs` is its wall-clock latency, blocking on I/O rather than on CPU.
N = 8
plan = [(f"tool_{i}", round(random.uniform(0.05, 0.30), 3)) for i in range(N)]
W = N                                          # one worker per independent subtask

# Sequential execution runs one subtask after another, so its critical path is
# the SUM of the latencies. Fanning the independent subtasks out to W workers
# runs them at once, so the parallel critical path is the MAX (the straggler).
serial_path   = sum(secs for _, secs in plan)
parallel_path = max(secs for _, secs in plan)
ideal_speedup = serial_path / parallel_path    # sum-over-max for independent work

# Amdahl-style ceiling: a real plan is part parallelizable (the fan-out) and
# part serial (the planner's reason->act->observe spine that cannot be skipped).
p = 0.85                                        # parallelizable fraction
amdahl_speedup = 1.0 / ((1.0 - p) + p / W)      # theoretical ceiling at W workers

print(f"subtasks N              : {N}")
print(f"workers  W              : {W}")
print(f"sum of subtask latency  : {serial_path:.3f} s  (critical path if serial)")
print(f"slowest single subtask  : {parallel_path:.3f} s  (critical path if parallel)")
print(f"ideal fan-out speedup    : {ideal_speedup:.2f}x  (independent sub-tasks)")
print(f"Amdahl ceiling (p={p})  : {amdahl_speedup:.2f}x  at W={W} workers")

subtasks N              : 8
workers  W              : 8
sum of subtask latency  : 1.066 s  (critical path if serial)
slowest single subtask  : 0.213 s  (critical path if parallel)
ideal fan-out speedup    : 5.00x  (independent sub-tasks)
Amdahl ceiling (p=0.85)  : 3.90x  at W=8 workers

5. Shared Memory and Interoperation Protocols Advanced

Parallel workers are only useful if they can share what they learn. A researcher worker that finds a key fact must make it visible to the critic worker that checks the final answer, and the supervisor must see every worker's result to plan the next round. Agentic systems therefore need shared memory: a state store the agents read from and write to, ranging from a simple shared scratchpad in the supervisor's context, to a blackboard that workers post intermediate findings on, to a persistent vector store of accumulated knowledge that is itself the retrieval index of Section 40.4. This is the shared-state coordination of Chapter 32, and it carries the same hazards as any distributed shared state: concurrent writes need ordering, stale reads mislead, and unbounded growth of the shared context blows the token budget. The engine that owns the shared memory must enforce consistency exactly as the coordination layer of Chapter 2 demands.

For agents and tools to interoperate at all, they must agree on protocols. Two are consolidating the field. The Model Context Protocol (MCP) standardizes how an agent connects to tools and data sources: a tool server advertises its functions and resources over a uniform interface, and any MCP-speaking agent can call them without bespoke glue, turning the tool layer of Section 2 into a pluggable, discoverable ecosystem. Agent-to-Agent (A2A) protocols standardize the complementary direction: how one agent delegates a task to another, exchanges intermediate state, and reports results, so that a supervisor can fan work out to workers that may run in different processes, on different machines, or even at different organizations. Together MCP and A2A do for agentic systems what the communication primitives of Chapter 4 did for parallel training: they define the interfaces over which distributed intelligence is exchanged, so that the orchestration engine can treat agents and tools as interchangeable, network-addressable participants.

We have now turned a single language-model call into a fleet of cooperating agents and tool services: a loop that reasons and acts, tool calls dispatched as distributed RPCs, a supervisor that fans independent work out to parallel workers, a durable engine that keeps the plan alive through failure, shared memory that lets the workers learn from each other, and protocols that let any agent call any tool. Every one of those agents and tool calls ultimately bottoms out in a language-model forward pass, served from a distributed inference fleet. That fleet, the engine every agent in this section depends on, is the subject of Section 40.7.