Where the Tokens Actually Go in Long-Context Inference

A context window is a capacity limit, not a promise that every included token will be used equally well. Two constraints appear as prompts grow: the serving system must hold and move more key-value state, and the model must still locate the evidence that matters. Long-context evaluations suggest that second constraint bites well before the advertised limit.

The KV-cache calculation

For a decoder-only transformer, an approximate per-sequence KV-cache footprint is:

2 × layers × KV heads × head dimension × tokens × bytes per value

The leading 2 represents keys and values. Consider an illustrative 70B-class grouped-query-attention configuration:

Input	Assumption
Layers	64
KV heads	8
Head dimension	128
Context	128,000 tokens
Storage	FP16, 2 bytes

The result is 33,554,432,000 bytes: 33.55 decimal GB, or 31.25 GiB, for one sequence. That is not the entire serving footprint. Model weights, activations, allocator overhead, batching, and framework metadata sit beside it. Tensor parallelism may shard the cache, while quantisation or eviction may reduce it.

The important operational point is linear growth: double the retained tokens and this component doubles. During autoregressive decoding, those retained keys and values are repeatedly accessed. PagedAttention addresses fragmentation and sharing in serving systems, but it does not make the underlying information free.

Capacity is not retrieval

Even when a prompt fits, the model can fail to use the right passage. Lost in the Middle showed position-dependent performance. That result should prevent a category error—treating “accepted 128k tokens” as “recalled any fact from 128k tokens with equal fidelity.”

This is why long context and retrieval-augmented generation are complements in many systems. Retrieval narrows the evidence set; the context window gives the model room for selected evidence, instructions, history, and output.

The retrieval layer has its own failure modes. A high-scoring passage may be related without containing the required evidence, as discussed in Why RAG Fails When the Chunk Score Is High. Evaluation must therefore label the evidence required for each question rather than use relevance scores as a proxy for correctness.

What sparse KV methods trade

StreamingLLM retains initial attention sinks plus a recent window to support streaming. H2O and SnapKV use attention-derived importance to retain a smaller subset. These approaches can reduce memory pressure, but the decision is an eviction policy: removed information cannot later contribute in the same way.

A production decision needs task-level evaluation, not only tokens per second:

1. Define the evidence a correct answer must use.
2. Vary evidence position and total context length.
3. Measure answer correctness and citation support.
4. Record time to first token, inter-token latency, and memory per active sequence.
5. Repeat with the proposed cache policy and realistic concurrency.

Model selection changes the same economics. Fewer layers, fewer KV heads, shorter retained histories, and lower-precision storage can all change the memory envelope. Which Model Should You Run in Production? treats that as a routing decision: the goal is not the largest available window, but the smallest model-and-context combination that meets the task's evidence requirements.

At serving time, report assumptions alongside results. “128k test” is insufficient without the exact model revision, cache precision, batch policy, tensor-parallel configuration, prompt-length distribution, output length, and whether prefix caching was warm. Those variables can change capacity and latency enough to reverse a comparison.

Long context is most useful when the system is explicit about what must remain available. Measure memory per sequence, retrieval success, and evidence use separately. A bigger window removes one limit; it does not remove the need for information architecture.