Prompt Caching: What It Saves and What Breaks It

Prompt caching stores the processed form of the beginning of a prompt, so that a later request starting with the same text can reuse it instead of paying to process it again. It is one of the few cost reductions that does not change what the model says. It also has one strict condition: the repeated text must match exactly, from the start, up to the point where the cache is marked.

Most teams either never turn it on or turn it on and wonder why it saves nothing. Both usually come from the same misunderstanding about what counts as the same prompt.

What is being reused

A request to a language model is processed in two parts. The prompt is read in parallel, and the attention states for every prompt token are computed and kept in the KV cache. Only then does generation begin, one token at a time. We covered that cache in what the KV cache actually costs you: the work of reading the prompt is substantial, and it has to be repeated for every request unless something keeps it.

Prompt caching keeps it. When a new request begins with the same token sequence, the serving system can load the stored states for that prefix and begin work at the first token that differs. The saving is largest when the shared prefix is long and the varying part is short.

The idea predates any one provider. The Prompt Cache paper proposes storing attention states for frequently reused text segments, such as system messages, templates and context documents, and reusing them when they appear in later prompts. The authors report latency improvements from 8 times on GPU-based inference to 60 times on CPU-based inference in their setups, with the largest gains on longer prompts such as document question answering. Those are their results on their configurations, and a production service will show different numbers, but the direction is consistent: the longer and more repeated the shared text, the more there is to save.

The rule that decides everything

Caching works on prefixes, not on content. The API documentation describes the rule plainly: cache hits require "100 percent identical prompt segments, including all text and images up to and including the block marked with cache control." The order in which the prompt is assembled matters too. Anthropic's documentation lists the prefix as tools, then system, then messages, and says that a change at one level invalidates that level and everything after it.

That has consequences that are easy to miss:

• Editing a tool definition invalidates everything. A renamed parameter or a reworded description means the tools section no longer matches, so the system prompt and messages that follow are recomputed too.
• A timestamp at the top of the system prompt defeats the cache. Something that changes every request, placed early, makes every request a miss.
• A user's name in the first line is the same problem. The shared part is broken at the first personal detail.
• Whitespace is content. A formatting change in a template that nobody thought was meaningful is a cache miss.

The practical discipline is to write prompts so the stable material comes first and the variable material comes last, and to keep the stable part byte-for-byte identical across requests. A team that treats its system prompt as a living document that gets edited daily will see its cache hit rate fall without any visible error.

What it costs and what it saves

Anthropic's documentation gives the pricing as multiples of the base input price. For most current models, writing an entry to the default five-minute cache costs 1.25 times the base input price, and reading an entry back costs 0.1 times the base price. The one-hour option is more expensive to write, at 2 times the base price, for workloads that repeat less often than every few minutes. The documentation notes per-model exceptions, so check the current table before doing sums.

The arithmetic is simple. Suppose a shared prefix is 10,000 tokens and it is reused across 100 requests within the cache lifetime. Without caching, you pay the base price for 1,000,000 tokens of that prefix. With caching, you pay the write premium once, 1.25 times the base price for 10,000 tokens, and the read price 99 times, at 0.1 times the base price. The total is about 12,600 token-equivalents against 100,000, roughly an eighth of the cost for the repeated part.

Three things reduce that saving in practice:

1. Low reuse. If each prefix is used once, you pay the write premium and get nothing back. Caching is a bet on repetition.
2. Expiry between requests. A five-minute lifetime that refreshes on use is generous for busy services and useless for a tool called twice a day.
3. Prefixes below the minimum length. The documentation says that prompts shorter than a model-specific threshold are not cached, and that no error is returned. The minimum varies by model, from a few hundred tokens to several thousand, so a short system prompt may never be cached at all.

How to know whether it worked

The provider returns usage counters with each response, and they are the only reliable check. The documentation describes two fields, one for tokens written to the cache and one for tokens read from it. If both are zero, the prompt was not cached, whatever you intended.

Log those fields per request, along with the prompt version. Then the question "did the change we made last week break caching?" has an answer, and it is usually a much faster one than the question "why did the bill rise?"

Measure at the level you actually serve. A cache hit rate of 90 percent on a development script says little about a production service with many different system prompts, users and tools, each with its own prefix. Each distinct prefix needs its own cache, and the number of distinct prefixes is often larger than people assume.

A useful exercise is to write the prompt out in its assembled order, as the provider receives it, and mark where the first difference between two consecutive requests occurs. Everything before that point can be cached, and everything after it is recomputed. Teams often find that the first difference sits far earlier than they expected, usually inside a template that interpolates a value nobody thought of as variable.

Caching and the other ways to save

Prompt caching sits alongside the other levers in what it costs to run an LLM per token. It does not change which model you use or how it is quantized, and it does not change the output. It only avoids recomputing text you have already processed.

It also interacts with serving. A shared prefix stored once can serve many requests, which is the same idea behind memory sharing in PagedAttention. Providers that offer caching are doing that sharing across customers' requests in ways their documentation describes; you control the part you can see, which is the prompt.

Designing a prompt for the cache

A simple layout works for most applications:

1. Tool definitions, which change least and should almost never be edited casually.
2. The system prompt, with role, policies and output format.
3. Reference material, such as documents, policies or examples that many requests share.
4. The conversation so far, which grows but is shared with the previous request.
5. The current user message and anything that varies, last.

Two rules follow. Do not mix per-request values into the first four sections. And when you must change one of them, do it deliberately, in a release, and watch the usage counters afterwards, rather than letting small edits accumulate into a cache that rarely hits.

What not to expect

Caching is not a memory feature. It does not make the model remember a user across sessions, and a stored prefix is not a conversation history. It is an optimisation over a text that is already fixed. If your application needs the model to know something from last week, that belongs in retrieval or in stored state, not in a cache that expires in minutes.

It is also not a guarantee of cheaper output. Input tokens may become cheaper while output tokens, which are generated one at a time and are the slower part of a response, cost the same. Read the bill by category before concluding the change worked.

A checklist

• Keep the first part of the prompt stable and identical across requests.
• Move every per-request value to the end.
• Check that the prefix exceeds the model's minimum cacheable length.
• Log cache writes and reads per request.
• Decide the cache lifetime from how often the prefix repeats, not from habit.
• Re-measure after every change to tools or the system prompt.

What we do not know

We do not know how long the current pricing structure will last, or how providers will share caches across customers in the future. The published results that favour caching come from specific setups, and the saving in your application depends on how often your prefixes repeat and how stable they are, which only your logs can tell you. The honest starting point is to measure your own hit rate for a week before deciding how much it matters.