The Legibility Problem in Autonomous Agents

An agent that produces the right answer through a process nobody can reconstruct is not a solved problem. It is an unaudited one.

This distinction stops being academic the moment a human is accountable for what the agent does. At that point the question is no longer "is it accurate?" but "when it is wrong, can I establish what happened, and can I do something before it matters?" Most agent architectures answer the first question and quietly fail the second.

Aggregate accuracy, per-case accountability

The mismatch is structural. Accuracy is a property of a distribution; accountability is assigned to individual cases.

Suppose an agent handles a support queue at 97% accuracy. That is a good number. It also means three cases in every hundred are wrong, and those three arrive without labels. Nothing distinguishes them at the time. If the person answerable for the queue cannot reconstruct any individual decision, then every output has to be treated as unverified — which is roughly the position they were in before automating it.

The value of the automation was supposed to be that a human stops reading every case. That only works if, when a case surfaces later, its history can be recovered. Otherwise you have not removed the review step; you have deferred it and made it harder.

Legibility is not explainability

These get conflated and they are different properties with different reliability.

	Explanation	Trace
Produced	After the fact, on request	During execution, as a side effect
By	The model, as generated text	The runtime, as a record
Can be plausible but false	Yes	No
Survives the model being replaced	No	Yes

An explanation is an artefact the model writes about itself. It can be fluent, internally consistent, and unrelated to the computation that produced the answer. This is the failure mode that makes chain-of-thought unreliable as an audit record: the text is not a description of what the network did, it is another generation conditioned on the same context.

A trace is different in kind. It records events that actually occurred — this tool was invoked, with these arguments, and returned this. It is boring, it is verifiable, and it does not depend on the model being honest about itself, because the model is not the author.

The practical rule: treat generated reasoning as a debugging hint and never as evidence.

What legibility costs

Legible architectures are frequently less capable, and pretending otherwise makes the trade impossible to reason about.

Forcing an agent to commit to an explicit plan before acting constrains it to a decomposition chosen up front, which is often worse than one discovered mid-task. Requiring every tool call to carry a structured justification adds tokens to every step and latency to every turn. Checkpointing for human approval converts a two-second run into one bounded by how fast a person responds.

Design choice	What it buys	What it costs
Explicit plan before execution	A comparable artefact per run	Locks in an early, sometimes worse decomposition
Structured tool-call records	Auditable, model-independent history	Tokens and latency on every step
Human checkpoints on side effects	Intervention before damage	Run time bounded by human response
Constrained action space	Smaller failure surface	Tasks outside the space simply cannot be done

None of these are free, and the right answer depends entirely on whether anyone is accountable for individual outputs. For a task where nobody is — internal drafting, exploratory search, anything where the human reads the result anyway — the trade is not worth making. The cost is real and the benefit is zero.

This is the same shape of decision as choosing between a small specialised model and a frontier one: the correct choice follows from the constraints of the deployment, not from which option is better in the abstract.

Where the checkpoints go

The instinct is to checkpoint the reasoning. It is the wrong place.

Pausing an agent between internal steps produces a stream of approvals a supervisor cannot meaningfully evaluate, because the individual step rarely carries enough context to judge. What follows is approval fatigue: the human clicks through, and a supervisor who approves everything is not supervising. The checkpoint has become theatre while adding latency.

The useful boundary is the point where an action becomes externally visible or expensive to reverse:

• sending a message to someone outside the system
• writing to a system of record
• moving money
• deleting anything

These are few, they are individually consequential, and a human can actually judge them. A run with four hundred internal steps and three external actions needs three checkpoints, not four hundred.

What does not work

Several approaches look reasonable and do not survive contact with a real system.

Logging everything. Capturing full prompts and completions at every step produces volume nobody reads. The failure is not storage cost — it is that an unstructured pile is not searchable in the way an investigation needs. "Which runs called the refund tool with an amount over £500" is answerable against structured records and effectively unanswerable against a transcript dump.

Asking the model to rate its own confidence. Self-reported confidence is generated text with the same reliability problem as generated reasoning. It correlates with fluency more reliably than with correctness, which makes it worse than useless as a routing signal — the confident wrong answers are exactly the ones it fails to flag.

Reconstructing intent after the fact. Feeding a completed run back to a model and asking what happened produces a narrative fitted to a known outcome. It reads well and encodes hindsight. If the trace was not captured during execution, it cannot be recovered afterwards.

What we do not know

The open problem is whether this scales past a single agent.

The reasoning above assumes one agent, one run, one supervisor who can hold the whole thing in their head. Once agents call other agents, the trace becomes a tree, and it is genuinely unclear what a human is supposed to read. Flattening it loses the causal structure that made it useful; presenting it whole exceeds what anyone will actually review. We do not have a good answer, and we are sceptical of anyone who claims a clean one — the honest position is that multi-agent legibility is unsolved rather than merely unimplemented.

There is a second uncertainty worth naming. Everything here assumes the supervisor is competent to judge the action when shown it. For refunds and messages that holds. For an agent operating in a domain where the human is not the expert, a checkpoint may only manufacture the appearance of oversight. That is a harder problem than instrumentation, and instrumentation will not fix it.

The practical position

Legibility is not a feature to add once an agent works. It is a property of the architecture, and retrofitting it means rebuilding the parts that make decisions.

The decision to make early is narrow: is anyone accountable for individual outputs? If yes, the agent needs execution traces rather than generated explanations, checkpoints at side effects rather than at reasoning steps, and an action space small enough to enumerate. Accept that this costs capability and latency, the same way a training cluster's real constraints are power and interconnect rather than GPU count — the binding constraint is rarely the one in the headline.

If nobody is accountable, skip all of it. The overhead is real and you would be buying nothing.