What Speculative Decoding Actually Saves

Speculative decoding makes a large language model generate text faster without changing what it says. A small, cheap draft model guesses the next several tokens. The large model then checks all of those guesses at once, in a single forward pass, keeps the ones it agrees with, and carries on from the first one it would not have chosen. When the guesses are good, you get several tokens for the price of one pass through the large model.

The reason this works, and the reason it is worth knowing about, is that generating text one token at a time leaves most of a GPU's arithmetic capacity idle. Speculative decoding spends that spare capacity to skip steps.

Why generation is slow in the first place

A language model produces one token per forward pass, and each pass has to read every weight in the model from memory. Reading is the bottleneck, not arithmetic. A GPU can do far more multiplications per second than the memory system can feed it numbers, so during decoding the arithmetic units spend most of their time waiting.

We covered the same effect from the memory side in what the KV cache actually costs you and from the precision side in what quantization actually costs you. They all point at one fact: decoding is bound by memory traffic, so any technique that gets more done per read of the weights wins.

That also explains the trick. Checking five proposed tokens in one forward pass costs very little more than producing one, because the pass reads the weights once either way and the extra arithmetic fits in the idle capacity. Verification is nearly free. Guessing is where the cost lives.

How it works, step by step

1. Draft. A small model generates a short run of candidate tokens, say four or five, one after another. It is cheap because it is small.
2. Verify. The large model processes the prompt plus all the candidates in one forward pass. For each position, it computes the probabilities it would have assigned itself.
3. Accept or reject. Starting from the first candidate, each token is accepted if the large model agrees, and an acceptance rule decides this probabilistically so that the result matches what the large model would have produced.
4. Correct and continue. At the first rejected token, the large model supplies its own token for that position, and the process restarts from there. Even the worst case produces at least one correct token per large-model pass, so progress never stalls.

The original speculative decoding paper describes this as sampling faster without changing the distribution, and reports a two to three times speed-up on T5-XXL, using off-the-shelf models without retraining or architecture changes. The DeepMind paper on speculative sampling reports a two to two and a half times decoding speed-up on Chinchilla, a 70 billion parameter model, in a distributed setup, without compromising sample quality.

The one number that decides the speed-up

Everything hangs on how often the draft's guesses are accepted. Call the average number of tokens that survive per large-model pass the accepted length. As illustrative arithmetic, ignoring the draft's own cost:

Average tokens accepted per large-model pass	Large-model passes for 100 tokens	Rough speed-up
1.0 (every guess rejected)	100	none
2.0	50	about 2x
3.0	33	about 3x
4.0	25	about 4x

The draft is not free, so real speed-ups are smaller than the table suggests, but it shows the logic. A draft that is right most of the time lifts the speed-up; one that is rarely right adds work and gives nothing back.

Predictability decides acceptance. Code, structured output and boilerplate are easy to guess, because the next token is often obvious. Open-ended creative writing is hard, because many continuations are plausible. The same system can be much faster on one workload than another.

Why the output does not change

The accept-or-reject rule is designed so the final text has exactly the distribution the large model alone would have produced. The draft only affects how quickly tokens are produced. It cannot decide what the answer says, because every token that survives has been approved by the large model.

That is an important difference from other ways of speeding up inference, such as quantization or switching to a smaller model, which trade some quality for speed. Speculative decoding in its standard form trades extra computation for time. Details of floating-point behaviour can still make two runs differ slightly, as they can for any serving change, so the claim is statistical equivalence, not bit-for-bit identity.

Variants that remove the second model

Running a separate draft model has costs. It needs memory, it has to be a good match for the target, and it is one more thing to deploy and keep in step with the main model. Several methods avoid it.

Medusa. The Medusa paper adds extra decoding heads to the target model itself, each predicting a token further ahead, and checks the resulting candidates using tree-shaped attention. Its authors report over a 2.2x speed-up without compromising generation quality when only the new heads are trained, and 2.3x to 3.6x when the heads and backbone are trained together.

EAGLE. EAGLE drafts at the level of the model's internal features instead of its output tokens. The authors report a latency speed-up of 2.7x to 3.5x on LLaMA2-Chat 70B while maintaining the distribution of the generated text.

Method	Where the guesses come from	Extra training	Reported speed-up (authors' results)
Speculative decoding	A separate small model	None	2x to 3x on T5-XXL
Speculative sampling	A separate small model	None	2x to 2.5x on a 70B model
Medusa	Extra heads on the target model	Heads, optionally the backbone	Over 2.2x, up to 3.6x
EAGLE	A small module predicting internal features	The module	2.7x to 3.5x on a 70B model

These figures are each paper's own results on its own models, hardware and benchmarks. They are evidence that the technique can work, not a prediction for your system.

When it does not help

The draft is poor. If acceptance is low, you pay for drafting and verifying and keep little of it.

The server is already busy. The method uses spare arithmetic capacity. At large batch sizes, where a GPU is already doing a lot of useful work per read of the weights, the spare capacity shrinks and the gain with it. Speculative decoding helps most at low batch sizes, where latency matters most, and least on a saturated throughput-oriented server.

Memory is the constraint. A separate draft model needs room, and so does the larger working set. A system that is short of memory may do better to spend it on a larger KV cache and more concurrent requests.

The workload is unpredictable. If your traffic is mostly open-ended generation, expect smaller gains than the headline numbers.

Who should care

If you use a model through an API, the provider decides whether to use the technique, and you will notice it only as price and speed. If you host models yourself, it is one of the few ways to cut latency without giving up quality, and most modern serving engines support some form of it.

It matters most where latency, not throughput, is the problem: interactive assistants, code completion, and voice agents, where time to first token is part of a tight budget. In those settings a two to three times faster decode can be the difference between a response that feels live and one that does not.

How to test it on your own traffic

1. Capture a representative sample of real requests. Include the long, odd and hard ones, not just typical prompts.
2. Measure the baseline at your real batch size: tokens per second per user, time to first token, and total throughput.
3. Enable the method with a draft model or head that suits your target, and keep every other setting identical.
4. Record the acceptance rate and accepted length if your serving engine exposes them. They explain any result.
5. Compare at several batch sizes. The gain at a batch of one will not hold at a batch of sixty-four.
6. Compare outputs on your evaluation set. The statistical claim is strong, but confirm nothing regressed in your setup.

The result is a small table of speed-up against load, which is the version of the claim that applies to you.

What we do not know

We do not know how well these methods carry over to the longest contexts and to the newest model families, since most published results come from specific models and benchmarks. The effect of the technique on cost per token at production batch sizes is also less documented than its effect on latency at small ones. The honest summary is that the idea is sound and the published speed-ups are real, and the number you will get is one you have to measure.