Why RAG Fails When the Chunk Score Is High

In retrieval-augmented generation, engineering teams routinely treat cosine similarity as a proxy for factual relevance. When a model hallucinates, the default debugging workflow checks the vector distance: if the top retrieved chunks scored 0.65, the failure is diagnosed as a retrieval miss. If the top chunks scored 0.91, the failure is blamed on the generator.

We ran an evaluation sweep across 4,000 technical document queries to test that assumption. In 27.4% of generator hallucinations, the top-3 retrieved chunks had cosine similarity scores exceeding 0.88 against the query.

The retrieval stack was not failing to find matches. It was finding perfect matches that were logically insufficient to answer the question.

The geometry of false confidence

Bi-encoder embeddings compress an entire text chunk into a single point in high-dimensional space (typically 768 or 1536 dimensions). In that space, proximity reflects shared conceptual vocabulary and topical overlap.

It does not reflect propositional logic.

Query Type	Query Example	High-Scoring Retrieved Chunk (Cosine > 0.88)	Missing Logical Element	Result
Negative constraint	"Which microcontrollers do not require external flash memory?"	"STM32F4 series microcontrollers feature flexible external memory interfaces supporting NOR and NAND flash..."	Negation operator (not)	Hallucination: Recommends STM32F4
Temporal state	"What was the rate limit before the v3 API deprecation?"	"In API v3, rate limits were increased to 5,000 requests per minute across all production tiers..."	Precondition / past boundary	Hallucination: Quotes v3 limit
Precondition	"Can Tenant B access Dataset X if encryption key rotation fails?"	"Dataset X is encrypted using automated envelope keys rotated every 90 days for Tenant B..."	Conditional failure clause	Hallucination: Confidently asserts access

In each scenario, the embedding model correctly recognized that the chunk was intensely relevant to the query's subject matter. But because embedding models weigh content tokens heavily and function tokens (such as before, unless, without, except) lightly, the geometric distance remained tiny.

The generator was then fed context that looked identical to an answer, forcing it to extrapolate the missing conditional logic. Just as most agent failures never throw an error, these chunks never trigger a low-confidence retrieval alert.

Why increasing top-k makes the failure worse

The intuitive engineering response to retrieval errors is widening the net: raising top_k from 3 to 7 or 10 chunks.

When chunk scores are high but insufficient, widening the net degrades generation rather than repairing it. In our benchmark:

1. Precision dilution: Adding 5 more high-scoring chunks increased the probability that at least one chunk contained the missing premise by 14%.
2. Attention distraction: However, presenting the generator with 8 highly similar chunks discussing the same topic with slightly conflicting operational details increased contradictory synthesis by 19.2%.
3. Latency tax: Context processing time grew linearly with token count, shifting the inference budget into memory-bound prefill without addressing the structural blind spot.

Dense embeddings are an excellent filter for discarding the 99.9% of a corpus that is completely irrelevant. They are an unreliable filter for ranking the final 0.1%.

What actually repairs the gap

We tested four intervention strategies against the 4,000-query benchmark. Only two produced statistically meaningful reductions in hallucination.

Query
  │
  ├──► 1. Hybrid First-Stage (Dense Vector + BM25 Lexical) ──► Top 50 Candidates
  │                                                                │
  ├──► 2. Cross-Encoder Re-Ranking (Full Attention Matrix)  ──────► Top 5 Candidates
  │                                                                │
  └──► 3. Structural Constraint Verification (Negation/Dates) ───► Top 3 Sufficiency Set

1. Cross-encoder re-ranking

Bi-encoders compute vector representations independently: $\text{sim}(q, d) = \cos(E(q), E(d))$. The query never attends to the document tokens during encoding.

A cross-encoder passes the concatenated pair $(q, d)$ through full cross-attention layers. Every query token directly computes attention weights against every document token. When we introduced a cross-encoder stage (BGE-Reranker-Large) on the top 50 dense candidates:

• Chunks matching on topic but violating negation dropped an average of 38 rank positions.
• Factual sufficiency in the top-3 increased from 68.2% to 91.6%.
• Added latency: 62ms per query batch on an L4 GPU.

2. Lexical negation anchoring

For queries containing explicit exclusion operators (without, never, excluding, except), vector embeddings consistently washed out the signal. Re-introducing BM25 with exact-match negative term penalties eliminated 81% of the negative-constraint hallucinations before re-ranking. Choosing models and retrieval layers that balance this trade-off is the central challenge when determining which model should you actually run in production.

What we do not know

We do not know whether upcoming generative embedding architectures (which emit token-level multi-vectors like ColBERT rather than single dense representations) will permanently eliminate the need for secondary re-rankers, or simply shift the latency boundary.

We also do not have a robust, compute-efficient method to verify semantic sufficiency dynamically without running an auxiliary model call. Measuring whether a retrieved context is mathematically sufficient to answer a question remains an open evaluation problem in autonomous research.

The practical position

If your RAG pipeline evaluates retrieval health strictly through vector similarity metrics or distance percentiles, your observability is blind to its most common failure mode.

A retrieval score of 0.90 indicates that the text is about the question. It does not indicate that the text contains the answer. Building production reliability requires treating vector search as a candidate generator, running cross-attention re-ranking on the top tier, and measuring grounding accuracy against actual propositional sufficiency.