The Hidden Latency Cost of Stateful Multi-Agent Loops

“Add another agent” sounds like adding another specialist. In an LLM system it also adds another inference boundary, another state-transfer decision, and often another copy of the conversation history.

The issue is not that multi-agent architectures are inherently bad. It is that their cost is hidden when diagrams show roles but omit the data passed between them.

A simple latency model

For a sequential graph, end-to-end latency is approximately:

Σ(prefill_i + decode_i + tool_i + queue_i) + orchestration overhead

Suppose a planner creates 1,000 tokens, a researcher adds 2,000, a critic adds 1,000, and an executor receives everything. If every role also receives a 4,000-token task packet, later calls repeatedly prefill a growing history. Exact time is provider-dependent, but the direction is not: serial stages add, and shared state expands downstream inputs.

Prefix caching can help when prompts share stable leading blocks. It helps less when roles use different system prompts, tool schemas, message ordering, or dynamically rewritten histories.

Long histories inherit the constraints described in Where the Tokens Actually Go in Long-Context Inference. A transcript fitting inside the advertised window is not proof that every role will find the relevant decision inside it.

When another role is justified

A separate agent is valuable when it creates a boundary the system actually needs:

• independent verification with a distinct rubric;
• security or permission isolation;
• parallel work on separable evidence;
• a specialised model or tool environment;
• an auditable handoff between accountable stages.

A theatrical “debate” between near-identical prompts is not automatically independent evidence. If both roles share the same model, context, and blind spot, agreement can be correlated rather than confirmatory.

Keep state typed and bounded

Passing the entire transcript is easy and often hard to inspect. Prefer an explicit state object:

Field	Purpose
objective	Stable success condition
constraints	Rules downstream stages must preserve
evidence	Source-linked observations
decisions	Chosen actions and responsible stage
open_questions	What remains unresolved
artifacts	Durable outputs referenced by identifier

Each stage should receive the smallest view needed for its job. Store large artifacts outside the prompt and pass references. Summaries should retain provenance so compression does not turn evidence into unattributed assertions.

Evaluate the graph, not the story

Trace every model and tool call as a span. Record input tokens, output tokens, cache status, latency, retries, state bytes, and outcome. Then remove roles one at a time. If deleting the critic changes neither supported accuracy nor failure detection, the critic is cost without demonstrated value. If no role survives that test, the task may not need an agent at all.

Compare one model with tools, a routed set of specialists, and the proposed collaborative loop on the same tasks and rubric. The right architecture is the smallest graph that meets the reliability requirement.

The trace also needs to be readable after a failure. The Legibility Problem in Autonomous Agents proposes a compact event contract for goals, evidence, decisions, tool actions, and approvals. Apply it at orchestration boundaries so an operator can identify which role introduced a bad assumption and which later roles merely repeated it.

Before shipping, run failure cases: contradictory tool output, stale state, a timed-out specialist, malformed structured output, and an approval that never arrives. Define whether each node retries, falls back, pauses, or terminates. Coordination is reliable only when those transitions are explicit.