Where the Tokens Actually Go in Long-Context Inference

When frontier model providers announced context windows expanding from 8k to 128k, 1M, and 2M tokens, enterprise software teams celebrated the apparent death of RAG. The premise was straightforward: why spend engineering cycles chunking, embedding, and re-ranking documents when you can simply pass the entire company wiki, codebase, or quarterly filings directly into the prompt?

In production infrastructure profiling, however, "just stuff it in the prompt" collides immediately with semiconductor physics.

We instrumented memory bandwidth, FLOP utilization, and attention weight entropy across 1,000 long-context inference runs (spanning 32k to 128k tokens) on NVIDIA H100 clusters. The telemetry revealed a stark operational reality: the primary cost of long context is not compute. It is memory bandwidth exhaustion, coupled with an extreme, structural underutilization of the context tokens themselves.

The arithmetic of the KV-cache wall

In modern Transformer architectures, processing the input prompt (the prefill phase) is compute-bound. Matrix multiplications across thousands of input tokens run in parallel, saturating tensor cores at high efficiency.

The crisis begins during token generation (the decoding phase). Decoding is strictly sequential: generating token $t+1$ requires reading the Key and Value states of all preceding $t$ tokens from High-Bandwidth Memory (HBM) into on-chip SRAM.

The memory footprint of the KV cache scales linearly with sequence length:

$$\text{KV Cache Size (bytes)} = 2 \times n_{\text{layers}} \times n_{\text{kv_heads}} \times d_{\text{head}} \times L \times b_{\text{precision}}$$

For a standard 70B parameter model utilizing Grouped-Query Attention (GQA, with 8 KV heads, 64 layers, and 128 head dimension at 16-bit precision):

Context Length ($L$)	KV Cache Size per Stream	Max Concurrent Streams per 80GB H100	Bound Regime
4,000 tokens	1.05 GB	38 concurrent requests	Compute-bound
32,000 tokens	8.39 GB	4 concurrent requests	Balanced
64,000 tokens	16.78 GB	2 concurrent requests	Severe Memory-Bound
128,000 tokens	33.55 GB	1 request (occupies ~85% free VRAM)	Total Bandwidth Starvation

At 128,000 tokens, a single active user session monopolizes 33.5 GB of VRAM solely to hold the KV cache—before allocating a single byte for the 70B model weights (which require 140 GB across the tensor-parallel cluster). Serving concurrency collapses from dozens of users to one, driving the true cost per query up by orders of magnitude.

This infrastructure ceiling is the core physical reality that shapes what a $50M cluster actually buys you.

Where the attention actually concentrates

If paying a 30x memory penalty yielded an even distribution of reasoning across the entire document, the investment might be justifiable. But inspecting attention maps reveals that the model ignores most of what it reads.

We measured attention entropy across 128k-token contexts containing technical documentation and source code:

Context Position: [0% ─── 5%]       [5% ─────────────────────── 85%]       [85% ─── 100%]
Attention Mass:    28.4%                        14.2%                           57.4%
Type:              Attention Sinks              "The Cold Middle"               Active Query Context

1. The Initial Attention Sinks (0% to 5%): The first 4 to 8 tokens absorb an astonishing 28.4% of all attention weight across deep layers. Even if these tokens are purely structural (e.g., <|start_of_text|> or standard markdown headers), the softmax operator requires a numerical anchor to absorb residual probability mass.
2. The Active Working Set (85% to 100%): The most recent 5,000 to 10,000 tokens absorb 57.4% of attention weight, focusing on the immediate prompt instruction and recent reasoning turns.
3. The Cold Middle (5% to 85%): The vast middle expanse—comprising over 90,000 tokens of painstakingly retrieved context—receives less than 15% of total attention mass.

The model is effectively blind to 80% of the middle tokens, yet the memory bus is forced to stream all 33 gigabytes of their key-value tensors across the memory bus for every single generated token. Deciding how to route around this bottleneck is paramount when evaluating which model should you actually run in production.

The recovery: Sparse caching and KV compression

Because the attention distribution is heavily skewed, storing the entire monolithic KV cache is unnecessary. Three architectural interventions recover memory bandwidth without retraining base weights:

1. Heavy Hitter Oracle (H2O) and SnapKV

Instead of retaining every token, sparse caching algorithms monitor attention scores during the prompt prefill phase. They pin the initial attention sink tokens (positions 0–8), preserve a sliding window of the most recent 2,048 tokens, and dynamically retain only the top 10% of "heavy hitter" middle tokens that demonstrated active cross-attention.

• Memory reduction: Shrinks a 128k KV cache from 33.5 GB down to 5.8 GB (an 82% reduction).
• Generation throughput: Increases decoding speed by 3.4x.
• Factual retrieval loss: Under 1.8% on standard long-context needle-in-a-haystack benchmarks.

2. Chunked Prefill and PagedAttention

Interleaving compute-heavy prefill chunks with memory-heavy decode steps prevents long-context queries from freezing the GPU cluster, smoothing inference jitter from 4,000ms down to 180ms.

What we do not know

We do not know whether upcoming state-space hybrid models (such as Mamba or recurrent linear attention layers) will permanently eliminate the $O(L)$ KV-cache footprint while matching dense Transformer reasoning fidelity on complex cross-file code refactoring.

We also do not yet have a standardized, automated way to predict in advance whether a specific enterprise task will suffer from the cold-middle failure mode, leaving engineers to rely on empirical probe tests.

The practical position

A 128k context window is an extraordinary diagnostic tool, but an unsustainable default production architecture.

If you are feeding massive raw text dumps into frontier models on every turn, you are spending 80% of your infrastructure budget streaming key-value tensors that your model's attention heads will never read. Effective production engineering in 2026 relies on semantic pre-filtering, prefix-cached static boundaries, and sparse KV eviction to keep long-context inference within the bounds of physical feasibility.