The Legibility Problem in Autonomous Agents

An agent is not legible because it can explain itself after the fact. It is legible when an operator can reconstruct what actually happened from durable records tied to execution.

That distinction matters once an agent can call tools, modify records, send messages, or make decisions across many steps. A polished narrative is not an audit trail.

The execution contract

Every material step should emit a structured event. A minimum event can include:

Field	What it answers
run_id and step_id	Which execution and sequence?
goal	What outcome was being pursued?
input_refs	Which versioned inputs entered the step?
evidence	Which observations supported the decision?
decision	What choice was made?
tool_action	What external operation was attempted?
result_ref	What changed or was returned?
uncertainty	What remained unknown?
policy_checks	Which constraints passed or failed?
approval	Was a human decision required and received?
timing and cost	How long and how expensive was the step?

References are often better than copying full payloads. They preserve provenance while reducing duplicated sensitive data. Each reference needs a stable version or content hash; otherwise the trace can point to something that later changes.

Observe actions, not hidden prose

ReAct helped establish a useful pattern of interleaving reasoning and action, but production observability should not depend on storing unrestricted internal reasoning. It should capture externally meaningful decisions, evidence, tool parameters, responses, and state transitions.

This boundary is both safer and more useful. Operators usually need to know why a refund was issued, which policy text was retrieved, which customer record changed, and whether approval existed—not an unbounded narration of token generation.

For retrieval-grounded decisions, store the evidence identifiers and versions. Why RAG Fails When the Chunk Score Is High explains why a similarity score is not enough to establish support.

Traces need decision boundaries

A trace becomes unreadable if every low-level event has equal weight. Mark the moments that change the plan, cross a policy boundary, modify external state, or escalate uncertainty. Those checkpoints create a reviewable skeleton while detailed spans remain available for diagnosis.

Human oversight should attach to these boundaries. A high-risk action can require approval; a low-confidence retrieval can trigger escalation; a repeated tool failure can stop the run. The trace should show both the rule and its outcome.

Test whether the system is actually legible

Give an operator a failed run and ask them to determine:

1. the intended goal and constraints;
2. the first point where execution diverged;
3. the evidence available at that point;
4. every external side effect;
5. whether recovery or rollback is possible.

Measure time to diagnosis and agreement between reviewers. If experienced operators reach different explanations, logging more text may not help; the event model probably lacks a stable concept such as decision, evidence, or state version.

Legibility is infrastructure for control. It makes evaluation, incident response, policy enforcement, and improvement possible because the organisation can connect outcomes to the decisions that produced them.

In multi-role systems, give every event the producing role and parent span. The state and latency costs described in The Hidden Latency Cost of Stateful Multi-Agent Loops become much easier to control when the trace exposes which node added context, retried, or changed the plan.