Your Context Window Is Smaller Than Advertised

The context window on a model card is the largest input the model will accept. It is not the length the model reliably uses, and treating the two as the same number is behind a quiet class of failures in retrieval systems and long-document tools.

The distinction sounds pedantic until a system that works on ten pages starts missing facts on eighty, while every request still succeeds and nothing reports an error.

Two numbers, not one

	Advertised context	Effective context
Defined by	The largest input accepted	The longest input that still performs the task
Where it comes from	The model card	Measurement on your task
Stable across tasks	Yes	No
What it tells you	What will not error	What will work

The effective figure is task-dependent. Pulling one fact out of a long document is easier than following a chain of references across it, and a model can be at ceiling on the first while struggling with the second at the same length.

The easy test flatters everyone

Most long-context marketing rests on needle-in-a-haystack: plant one fact in a long filler document and ask for it back. It is a real capability and a low bar.

The RULER benchmark was built because that bar is low. It extends the needle test with multiple needles, multi-hop tracing and aggregation, then evaluates long-context models across lengths. Its central finding: models that score almost perfectly on the plain needle test still show large performance drops as context grows, and of 17 models evaluated that all claimed 32K tokens or more, only about half kept satisfactory performance at 32K.

These results are from models evaluated in 2024, and newer models have improved. The lasting point is methodological. A single-needle score tells you the model can find one thing, and very little about whether it can reason over everything you give it.

Position matters as much as length

The second finding is about where information sits, not how much of it there is.

In Lost in the Middle, researchers moved the answer-bearing document around within a set of retrieved documents. Accuracy was highest when the relevant passage was at the start or end of the input and dropped when it sat in the middle, including for models designed for long contexts. The shape is the U-curve in the figure above.

This has an uncomfortable implication for retrieval pipelines that concatenate ranked chunks. Ranking puts the best chunk first, which helps. It also means the second- through ninth-best chunks land in precisely the region the model reads least carefully, which is part of why a high chunk score does not guarantee a correct answer.

What this changes in practice

Retrieve less, and better. Doubling the number of chunks does not double the evidence the model uses. Past a point it adds material to the middle of the prompt, where it is least likely to be used, and it costs latency and tokens on every call. Where those tokens go is its own argument for keeping prompts short.

Put what matters at the edges. Instructions and the single most important piece of evidence belong at the start or end, not buried between documents.

Measure your own effective length. Take your real documents, plant the answer at several positions and several total lengths, and record where accuracy falls off. That number is the budget. It usually takes an afternoon and replaces a guess with a measurement, the same method that settles which model to run.

A test you can run in an afternoon

The measurement is a small grid. Hold the question and the answer-bearing passage fixed, and vary two things:

Vary	Values to try	Why
Position of the passage	Start, a quarter in, the middle, three quarters, the end	Exposes the middle-of-prompt dip
Total prompt length	Your typical length, then double, then four times	Finds where accuracy starts to fall

Pad with your own real documents rather than generated filler, since real documents are more repetitive and harder to search. Twenty to thirty questions per cell is enough to see the shape. The length at which the middle positions fall away is your working budget.

What does not work

Upgrading to a larger window to fix missed facts. If the failure comes from position or task difficulty, a bigger ceiling changes nothing about how the existing input is read.

Trusting a vendor needle chart. It measures the easiest task at the most favourable setting, usually with the needle placed and phrased to be found.

Assuming a newer model fixed it. It may have. Without a test at your length on your task, you do not know, and the failure is silent when it happens.

You may not need to worry about this

If your prompts are a few thousand tokens, instructions plus a short document or a handful of retrieved passages, none of this is likely to bite. The degradation described above appears as inputs grow long relative to what the model handles well. Short prompts sit comfortably inside the effective window of any current model.

The concern starts with whole-document and whole-codebase workflows, and with retrieval systems that fill the window because it is there.

What we do not know

We do not know the current effective length of any particular production model on any particular task, and neither does a model card. The evaluations cited here are from 2023 and 2024, and later models may have narrowed both gaps considerably.

We are also unsure how well synthetic benchmarks predict behaviour on real documents, which are more repetitive and more structured than generated filler. The honest position is that this is a thing to measure, not a figure to quote.

The figure in this note may be reused with attribution and a link to this page.