What It Costs to Run an LLM, Per Token
Cost per token is GPU price divided by useful tokens per second. Utilisation, batching, prompt length and model choice move it more than the hourly rate.
By Cogniq Labs ResearchEvidence policy
The cost of running a language model comes down to one division: what the hardware costs per hour, over how many useful tokens it produces per hour. The top of that fraction is a price you can look up. The bottom is a number you have to earn, and it is the part that moves cost by factors of two, five or ten.
Most discussion of LLM pricing is about the top. This note is about the bottom.
The formula
Cost per million tokens is:
cost per million tokens = (hardware cost per hour ÷ 3,600) ÷ tokens per second × 1,000,000
As illustrative arithmetic, suppose a GPU costs $2.50 an hour and, with good batching, serves 1,000 tokens per second across all its requests. The cost is about $0.69 per million tokens.
Now suppose the same GPU receives so little traffic that it averages 250 tokens per second. The hardware bill has not changed, but the cost is about $2.78 per million tokens, four times higher. Nothing about the model or the price of the GPU changed. Only utilisation did.
Those figures are hypothetical, chosen to show the mechanism. The point is that the denominator is not a spec-sheet number. It depends on how your traffic arrives and how well your serving stack handles it.
Why input and output cost different amounts
A request has two phases with different economics.
Input tokens are processed in parallel. The whole prompt goes through the model at once, which keeps the GPU's arithmetic units busy. This phase is compute-bound and efficient.
Output tokens are produced one at a time. Every new token requires reading all the model's weights and the growing KV cache from memory, while doing very little arithmetic. This phase is memory-bound, and the GPU spends much of it waiting.
The paper Efficiently Scaling Transformer Inference illustrates the difference in its measurements on a 540 billion parameter model: it reports about 29 milliseconds per token at low batch size during generation, while processing input in large batches reached 76 percent of the hardware's peak arithmetic throughput. These are one paper's results on specific hardware, but they show the shape: parallel input processing uses the chip well, and one-token-at-a-time output does not.
This is why providers commonly charge more for output tokens than input tokens, and why a prompt that asks for a short answer is cheaper than one that asks for an essay.
What actually moves the number
| Lever | What it changes | Typical effect |
|---|---|---|
| Hardware price | The numerator | Real, but usually the smallest lever you control |
| Utilisation | How much of the day the GPU does useful work | Large: idle time raises cost per token directly |
| Batching | How many requests share each read of the weights | Large: more requests per read lowers cost per token |
| Memory efficiency | How many requests fit alongside the weights | Large: the KV cache limits concurrency |
| Prompt and output length | Tokens per request | Linear: half the tokens, roughly half the cost |
| Model size and type | Compute and memory per token | Large: see below |
| Precision | Bytes per weight | Moderate: see what quantization costs |
Batching deserves emphasis. Because decoding is limited by memory reads, serving more requests together costs little extra per read, so throughput rises much faster than cost. A server that handles requests one at a time can cost several times as much per token as one that batches well. That is also the reason a lightly used private deployment often loses to a shared API: the API provider's utilisation is higher than yours can be.
Memory management matters too. The PagedAttention paper describes how wasted KV cache memory limits batch size, and reports that reducing the waste allows more requests to be batched together and higher throughput. Better memory use is a cost reduction that needs no new hardware.
Model choice is a cost decision
A larger model costs more per token in two ways: it does more arithmetic per token, and it leaves less memory for the cache, so fewer requests fit. The reverse also holds, which is why small models are winning the fights that matter for narrow, repeatable tasks.
Architecture matters as well. A mixture-of-experts model is cheaper per token in compute, but not in memory, so it helps when compute is the constraint and hurts when memory is. And decoding techniques such as speculative decoding can cut latency at low load without changing the output.
The practical rule is to pick the smallest model that meets your quality bar on your own evaluation set, since a leaderboard will not tell you which model to run in production.
Latency and cost pull against each other
Batching lowers cost per token, but it also makes each user wait a little longer, because their request shares the GPU with others. Serving a single request at a time gives the fastest response and the worst cost. Serving a large batch gives the lowest cost and slower individual responses.
Where you sit on that curve is a product decision, not a technical one. A chat assistant, a voice agent and a nightly batch job each need a different point. Interactive features such as voice agents with a tight response budget cannot give up latency for cost, while work that nobody is waiting for, such as summarising documents overnight, should be batched as hard as possible. Separate the two kinds of traffic where you can, so the batch work does not drag down the interactive work and the interactive work does not waste the batch work's efficiency.
Reuse work you have already paid for
Many requests begin with the same text: a system prompt, tool definitions, a document everyone is asking about. Recomputing it for every request is wasted arithmetic. Serving systems can keep the processed form of a shared prefix and reuse it, and several hosted APIs discount repeated prefixes for the same reason. The rule is the one from the KV cache note: put stable content first, keep it identical across requests, and put what varies at the end.
The same logic applies one level up. If the same question is asked many times, caching the final answer costs nothing per request. Check first that a stale answer is acceptable, because a cache that serves out-of-date information has traded cost for correctness.
Do not send every request to the same model
Not every request needs the best model. The FrugalGPT paper groups ways of lowering cost into three kinds: adapting the prompt, approximating the model with a cheaper one, and cascading across models so that a cheap one answers first and a stronger one is called only when needed. The authors report that a learned cascade can match the performance of the best individual model, such as GPT-4, with up to 98 percent cost reduction on their tasks, or improve accuracy by 4 percent at the same cost.
Those are results on the paper's own benchmarks, and a cascade needs a way to judge whether the cheap answer is good enough, which is a hard problem in itself. But the structure is useful. Most real traffic is a mixture of easy and hard requests, and paying frontier-model prices for the easy ones is the most common source of avoidable cost.
Self-host or use an API?
The difference is who carries the utilisation risk.
| API | Self-hosted | |
|---|---|---|
| Pricing | Per token, you pay for what you use | Per hour, you pay whether busy or idle |
| Utilisation risk | The provider's | Yours |
| Cost at low or spiky volume | Usually lower | Often higher |
| Cost at high, steady volume | Can be higher | Often lower |
| Control, privacy, latency tuning | Limited | Full |
| Operations work | None | Considerable |
Self-hosting is the right call when traffic is high and predictable enough to keep GPUs busy, or when data, latency or control requirements rule out sending requests elsewhere. It is usually the wrong call because of a per-token price comparison made at an imagined 100 percent utilisation. Compute your own cost at the utilisation you will actually achieve, including the people who keep it running.
A worksheet
To estimate your real cost per request:
- Measure tokens per request, input and output separately, over a representative sample.
- Measure aggregate throughput at your real concurrency, in tokens per second.
- Compute cost per million tokens with the formula above, using the throughput you measured, not a peak figure.
- Multiply by tokens per request to get cost per request, and by monthly volume to get the bill.
- Add the costs that are not tokens: retrieval, storage, retries, evaluation, and engineering time.
- Look at the distribution, not the average. A few very long requests can dominate the bill.
- Repeat for each model and configuration you are considering, and compare cost against quality on your own task.
The result is a cost per successful outcome, which is the number that matters. A cheaper model that fails more often can cost more per resolved task than an expensive one that gets it right.
What we do not know
We do not offer a universal price per token, because it does not exist: it depends on hardware, model, traffic shape and the quality of the serving stack, all of which vary widely. We also do not know how quickly the price of inference will keep falling. The reliable method is to measure your own tokens, your own throughput and your own quality bar, and let the formula do the rest.
Sources
Frequently asked questions
How do you calculate the cost per token of running an LLM?
Divide the hourly cost of the hardware by the number of useful tokens it produces per hour. Cost per million tokens equals the hourly price divided by 3,600, divided by tokens per second, times one million. The denominator is the hard part: it depends on batching, utilisation and request shape, not on the GPU alone.
Why are output tokens more expensive than input tokens?
Input tokens are processed together in parallel, which uses the GPU's arithmetic efficiently. Output tokens are produced one at a time, and each one requires reading all the model weights and the growing KV cache from memory, so the hardware does less useful work per second. Providers commonly price output higher for this reason.
Is it cheaper to self-host an LLM than to use an API?
Only at sufficient and steady volume. A self-hosted GPU costs the same whether it is busy or idle, so low utilisation can make the cost per token higher than an API's. Self-hosting tends to win when traffic is high and predictable, or when privacy, latency or control requirements rule out an API.
How can I reduce LLM costs without hurting quality?
Shorten prompts, reuse shared prefixes through caching, route easy requests to smaller models, batch work that is not latency-sensitive and set output limits. FrugalGPT reports matching GPT-4's performance on its tasks with up to 98 percent lower cost by choosing which model to call per query.
What does GPU utilisation have to do with token cost?
The hourly price is fixed, so every idle second raises the cost of each token you do produce. A GPU running at a quarter of its achievable throughput costs four times as much per token as one running at its limit, which is why batching and traffic shaping often matter more than choosing cheaper hardware.
Related reading
What Speculative Decoding Actually Saves
Speculative decoding lets a small model draft tokens that a large one checks in a single pass. The output is unchanged, and the speed-up depends on one thing.
What Quantization Actually Costs You
Quantization stores a model's weights in fewer bits, so it fits in less memory and often runs faster. Where the quality goes, and how to measure it.
Mixture of Experts: Cheaper per Token, Not per GPU
A mixture-of-experts model runs only a few experts per token, so compute falls. Every expert still has to sit in memory. Where that trade pays off.