What the KV Cache Actually Costs You

A served language model has two big things in GPU memory. The weights are the famous one, and their size is printed on the model card. The other is the KV cache, which is rarely mentioned, grows with every token of every request, and on long prompts quietly decides how many users one GPU can serve.

What it is

To produce each new token, a transformer attends over every token that came before it. Without a cache, generating the hundredth token would mean recomputing the keys and values for the ninety-nine before it, and again for the hundred-and-first, and so on.

The KV cache stores those keys and values once and reuses them. It is the reason generation is fast. The price is memory, held for the whole life of the request.

Why it grows

The cache size is a product of five things:

Factor	Grows when
Layers	The model is deeper
Key-value heads	The attention design keeps more of them
Head dimension	The model is wider
Sequence length	The prompt and output get longer
Concurrent requests	More users are served at once

The first three are fixed by the model. The last two are set by your traffic, which is why the cache behaves so differently from the weights: the weights cost the same whether one person is using the model or a thousand, and the cache does not.

A worked example

The following is illustrative arithmetic for a hypothetical 70-billion-parameter model with 80 layers, 8 key-value heads of dimension 128, stored at 16-bit precision. It is not a measurement of any product.

Each token stores a key and a value in every layer and every key-value head:

2 × 80 layers × 8 heads × 128 dimensions × 2 bytes ≈ 320 KB per token

At an 8,000-token context that is roughly 2.6 GB for a single request. Thirty-two such requests at once need about 84 GB of cache, on top of roughly 140 GB of weights.

If the same model kept a separate key and value for each of 64 attention heads instead of sharing 8, every one of those figures would be eight times larger. That difference is not an implementation detail. It decides whether the deployment fits on the hardware at all.

Two phases: where the cache is built and where it is read

Serving a request happens in two phases with very different costs.

Prefill processes the whole prompt at once. Every prompt token's keys and values are computed in parallel and written into the cache. This phase is dominated by arithmetic, and it is what you wait for before the first token appears. A longer prompt means a longer prefill, which is why time to first token rises with prompt length.

Decode then generates one token at a time. Each step reads the model weights and the entire cache built so far, does a comparatively small amount of arithmetic, and appends one more entry. That makes decoding limited by memory bandwidth rather than compute. The multi-query attention paper names exactly this problem: incremental decoding is slow because of the memory-bandwidth cost of repeatedly loading the large key and value tensors.

So the cache costs you twice. It occupies memory that could hold other requests, and it has to be read in full for every token generated, so each new token of a long conversation is slower to produce than the last.

Why it sets the price

Memory left over after the weights determines how many requests can be batched together, and batch size is what turns a GPU into throughput. A bigger cache per request means fewer requests per GPU, and fewer requests per GPU means a higher cost for every generated token.

This is also why long prompts are disproportionately expensive to serve, and part of why the tokens in a long context are not free even when the model handles them well.

What shrinks it

Share the keys and values across heads. Multi-query attention keeps one set of keys and values for all heads, which its author proposed precisely because loading these tensors was the bottleneck in decoding. Grouped-query attention shares a set across each group of heads, and its authors report quality close to full multi-head attention at speed comparable to multi-query. The hypothetical model above already uses this; it is the reason the cache is eight times smaller than it would otherwise be.

Stop wasting the memory you have. Early serving systems reserved one contiguous block per request, sized for the longest output it might produce. PagedAttention, the technique behind vLLM, manages the cache in small fixed-size pages instead, the way an operating system manages memory. Its authors report near-zero waste, sharing of cache blocks within and across requests, and two to four times the throughput of the systems they compared against at the same latency.

Send less context. The cheapest token to cache is the one you never send. Retrieving a few relevant passages instead of whole documents cuts the cache along with everything else, and a long prompt is not guaranteed to help anyway, since models often miss what sits in the middle of it.

How to estimate it for your own model

You do not need to trust anyone's figure. The calculation needs five numbers, all of which appear in a model's published configuration:

You need	Where to find it
Number of layers	Usually listed as the number of hidden layers
Number of key-value heads	Listed separately in grouped-query models; otherwise equal to the attention heads
Head dimension	Listed directly, or hidden size divided by the number of attention heads
Bytes per value	2 for 16-bit, 1 for 8-bit cache storage
Tokens and requests	Your expected context length and concurrency

Multiply them:

cache bytes = 2 × layers × KV heads × head dimension × bytes per value × tokens × concurrent requests

Then compare the result with the GPU memory left after the weights, and keep a margin for activations and framework overhead. If the cache for your target context and concurrency does not fit, something has to give: shorter contexts, fewer concurrent requests, a smaller cache precision, more hardware, or a model with fewer key-value heads.

Doing this before choosing a model is cheap, and it regularly changes the choice.

Shared prefixes are memory you can reuse

Many requests to the same application begin with identical text: a system prompt, tool definitions, a policy document, a set of examples. Without sharing, every request stores its own copy of that prefix in the cache.

PagedAttention's block design allows cache blocks to be shared within and across requests, so an identical prefix can be stored once. Several hosted APIs now price repeated prompt prefixes at a discount for the same underlying reason.

The practical rule is about prompt order. Put stable content first and keep it byte-for-byte identical across requests, and put the parts that vary, such as the user's question and retrieved passages, at the end. A prefix that changes by one character at the start cannot be shared.

You may not need to think about this

If you call a model through a hosted API, the provider manages the cache and you pay per token. You will feel it only indirectly, as higher prices and slower responses on long prompts, which is reason enough to keep prompts lean but not a reason to learn the internals.

It matters when you serve models yourself, and especially when you choose between them. Two models with similar benchmark scores can differ several times over in how many concurrent users fit on the same GPU, which is one of the less visible reasons a smaller model often wins in production.

What we do not know

The figures above assume 16-bit storage and a specific configuration. Serving stacks increasingly quantise the cache to lower precision, which shrinks it further at some cost in quality, and how much quality is lost depends on the model and task in ways that published results do not settle. We would measure it on your workload rather than assume it is free.

The figure in this note may be reused with attribution and a link to this page.