Most Agent Failures Never Throw an Error

An agent that crashes is the cheap case. You get a stack trace, a timestamp, and something to fix.

The expensive failures are the ones where every component reports success. The retrieval returned documents. The tool call returned a valid response. The model produced fluent, well-formed, correctly structured output. Nothing anywhere in the pipeline is in a position to observe that the answer is wrong, and so nothing does.

The error rate is not the failure rate

These get treated as the same number and they are not close.

Your error rate measures execution: timeouts, exceptions, malformed responses, tool calls that threw. Your failure rate measures outcomes: how often the agent did the wrong thing. In a pipeline where a language model sits between the inputs and the output, the second number can be an order of magnitude larger than the first, because the model's core competency is producing plausible output from whatever it is given.

That competency is usually described as robustness. In this context it is the problem. A strict parser fed a malformed field throws. A model fed a malformed field infers what was probably meant and carries on, and it is right often enough that you stop checking.

Three failures that report success

Retrieval returning plausible-but-irrelevant context. The query embeds near documents that share vocabulary but not subject. The agent receives five documents, all topically adjacent, none actually answering the question, and writes a confident answer from them. Retrieval reports five hits. The correct behaviour, returning nothing, looks identical in the metrics to a failed search.

Schema drift. A tool response changes shape. A field is renamed, a type changes, something that was a list becomes a single object. The agent adapts. It does not fail, because adapting to messy input is what it is good at, and it produces an answer built on a field it has misread. This one is particularly nasty because it appears at deploy time on the tool's side, with no corresponding change on yours.

Silent truncation. Context runs long, and something gets dropped, usually the middle. The agent answers from what remains. Nothing reports that the input was incomplete, and the answer is not obviously deficient, it is just missing the part that would have changed it.

Why your evals do not catch this

	Static eval suite	Production
Inputs	Fixed, hand-chosen	Open, drifting
Correct answer	Known	Unknown, that is the point
Failure signature	Output differs from expected	Output looks fine
What it measures	Regression against past behaviour	Whether the thing works

An exact-match suite tells you whether today's system agrees with yesterday's on cases you thought of. That is genuinely useful, and it is regression testing, not failure detection. Silent failures happen on inputs nobody wrote a test for, and they produce output shaped correctly enough to pass a superficial comparison.

A 96% pass rate on a static suite is entirely consistent with a much worse failure rate in production, because the suite is not sampling the same distribution as your users.

What to instrument

The useful records are of things that actually happened, not of what the model says happened. This is the same distinction that makes execution traces trustworthy where generated explanations are not: the runtime is the honest narrator, and the model is not.

Worth recording:

• what was retrieved, and separately, what was actually used
• every tool call, with arguments and raw response
• every point where the agent selected among options
• input sizes at each boundary, so truncation is visible rather than inferred

The cheap invariants catch more than expected. An empty retrieval set followed by a confident answer is a detectable contradiction. A tool response missing a field the agent then references is detectable. An answer asserting something none of the retrieved sources support is detectable, imperfectly, and imperfect beats absent.

What does not work

A second model reviewing the first. The reviewer shares the failure modes of the model it reviews, and language models agree with fluent confident text, which is precisely the shape of a silent failure. It catches formatting problems and outright self-contradiction. It systematically misses wrong retrieved context, because it reads the same wrong context and finds the answer well supported.

Self-reported confidence. Confidence scores correlate with fluency more reliably than with correctness. The failures you most need flagged are the ones the model is most sure about.

Raising the retrieval threshold. The intuition is that stricter matching filters irrelevant documents. In practice it trades one silent failure for another: instead of answering from wrong context, the agent answers from no context, and a model given nothing will still produce something.

Logging everything. Full prompts and completions at every step produce volume nobody reads and, worse, volume nobody can query. "Which runs proceeded after an empty retrieval" is answerable against structured records and effectively unanswerable against a pile of transcripts.

What we do not know

There is no general method for detecting semantic failure without knowing the correct answer, and we are sceptical of claims otherwise, because a system that reliably identified wrong answers would also be a system that could produce right ones.

What exists is a set of partial, cheap checks that catch a fraction of cases, plus sampling by a human who knows the domain. How large that fraction is, we genuinely do not know, and it certainly varies by task in ways we cannot currently predict. The honest position is that this is a mitigation rather than a solution.

The second uncertainty is drift over time. A pipeline that behaves well at launch degrades as the underlying documents, tools, and models change beneath it, and the degradation is invisible by construction. We do not have a good answer for how often to resample, other than more often than feels necessary.

The practical position

Stop treating the error rate as a proxy for whether the system works. They measure different things, and the gap between them is where the cost lives.

Instrument the boundaries, write the cheap invariants, and sample real outputs by hand at a rate that feels excessive. Accept that a fraction of failures will only ever be found by a person who knows what the answer should have been, and budget for that person. As with choosing a model for a narrow production task, the binding constraint turns out not to be capability, but whether you can tell what the system is actually doing.