Why RAG Fails When the Chunk Score Is High

The top-ranked chunk can be about the right subject and still fail to answer the question. That is not a contradiction. Embedding similarity and evidential support are different properties.

Proximity is not entailment

Dense retrieval places semantically related text near a query in vector space. It is effective because relevant passages often share meaning and terminology. But a score does not certify that a passage contains the decisive date, exception, causal link, or comparison.

Consider the question, “Can plan A be cancelled after renewal?” A chunk describing plan A, cancellation, and renewal may rank highly while omitting the sentence that says renewals are non-refundable. It is topically relevant but evidentially incomplete.

Scores also have no portable absolute meaning. Changing the embedding model, normalisation, distance metric, corpus, or query expansion changes their distribution. Approximate indexes add one more variable, since they trade recall for speed without reporting what they missed.

Diagnose three separate stages

A useful evaluation keeps these questions apart:

1. Retrieval recall: Did the candidate set contain the required evidence?
2. Evidence use: Did the model base its claim on that evidence?
3. Answer quality: Was the final answer correct, complete, and appropriately qualified?

If the evidence never entered the candidate set, prompt changes cannot fix retrieval. If it was retrieved but ignored, ranking, context placement, or generation is the likely failure. If the answer cites the passage but overstates it, the problem is faithfulness.

Position effects can matter after retrieval. Where the Tokens Actually Go in Long-Context Inference separates raw context capacity from reliable evidence use. A relevant chunk that is technically present but consistently ignored should be counted as an end-to-end failure.

Build evidence-labelled test cases

For each representative question, store the minimal passages that support the answer and any passages that look relevant but do not. Then evaluate recall at several candidate depths. Add hard cases: negation, dates, tables, product versions, cross-document joins, and terminology that changed over time.

Late-interaction retrievers such as ColBERT preserve more token-level matching than a single-vector representation. Cross-encoders can rerank a smaller candidate set against the complete query. Neither removes the need for labelled evaluation.

Make the generator show its work

Require claim-level citations or evidence spans. Reject or qualify an answer when support is absent. Track unsupported claims separately from incorrect ones: a plausible answer without evidence is still unsafe in a retrieval-grounded system.

Longer context is not an automatic cure. Adding more chunks can bury the decisive passage, increase conflicts, and worsen position effects. Retrieve enough to preserve recall, rerank for support, and pass only the evidence the generator can reasonably inspect.

The operational lesson is simple: optimise for supported answers, not attractive similarity numbers.

Monitor drift after launch

A retrieval system changes when documents, embedding models, chunking rules, or user language change. Version all four. Keep a small stable evaluation suite for regression detection and a rolling sample of real, privacy-reviewed questions for distribution shift.

When a failure appears, save the query, expected evidence, retrieved candidates, ranks, document versions, and final answer. That record turns a vague complaint into a test case. It also makes it possible to tell whether the next fix improved retrieval or merely changed the wording.

For high-impact actions, connect retrieval confidence to control policy. The Legibility Problem in Autonomous Agents shows what the execution trace should preserve. Low evidence recall can trigger clarification or human review; it should not silently become model confidence.