Why vLLM Runs Out of Memory, and How to Fix It

Most vLLM out-of-memory errors are not caused by the model being too big. They happen because vLLM was asked to guarantee a context length the GPU cannot hold, and it refuses at startup. Set --max-model-len to the length you actually need. That one change fixes most of them.

The rest depends on when the failure happens. An error while loading weights, one at startup about the cache, a crash during warmup and slowdowns under load all have different causes and different fixes.

Find your error

When it happens	What you see	What it means	First fix
Loading weights	CUDA out of memory	The weights do not fit	Quantized model, or more GPUs with --tensor-parallel-size
Startup	No available memory for the cache blocks	Nothing left for the KV cache after the weights	Raise --gpu-memory-utilization, or free memory
Startup	KV cache is needed, which is larger than the available KV cache memory	One full-length request does not fit	Lower --max-model-len
Warmup	CUDA out of memory occurred when warming up sampler	Too much reserved, too little headroom	Lower --max-num-seqs or --gpu-memory-utilization
Under load	No crash; latency spikes	Requests are being preempted	More KV cache room, or fewer concurrent requests

Exact wording changes between releases, so match on the phrase and the stage rather than the whole line.

How vLLM divides the GPU

vLLM does not allocate memory as it goes. At startup it takes a fixed share of the GPU, set by --gpu-memory-utilization, which the engine arguments page describes as the fraction of GPU memory used for the model executor. The current documentation lists a default of 0.92, and defaults can differ between releases, so check yours.

Inside that share, memory goes in this order:

1. Model weights.
2. Activations, the working memory for a forward pass, which vLLM measures with a test run.
3. CUDA graphs, which the memory documentation notes take up extra GPU memory.
4. The KV cache, which gets whatever is left.

The cache comes last, and it is what requests actually need. We explained why it grows with every token in what the KV cache actually costs you. This order also explains something that alarms people: nvidia-smi shows the GPU nearly full with no traffic. vLLM reserved that memory on purpose.

A worked example

Take Llama 3.1 8B on a 24 GB GPU, in 16-bit precision. Its configuration has 32 layers, 8 key-value heads and a head dimension of 128. The following is arithmetic, not a measurement, and the activation figure is an assumption.

Item	Size
vLLM's share at 0.92 of 24 GB	≈ 22.1 GB
Weights, 8 billion parameters at 2 bytes	≈ 16 GB
Activations and CUDA graphs (assumed)	≈ 1.5 GB
Left for the KV cache	≈ 4.6 GB

Each token of cache takes 2 × 32 × 8 × 128 × 2 bytes, which is 131,072 bytes, or 128 KiB. So 4.6 GB holds about 35,000 tokens in total, shared between every request running at once.

Now the failure becomes obvious. The model supports a context of about 128,000 tokens. If --max-model-len is left unset, vLLM takes that value from the model config, and one request of that length needs about 16 GiB of cache. Only about 4.6 GB is available, so vLLM refuses to start, even though the model fits easily.

A public report shows the same pattern: a maximum sequence length of 116,160 needed 4.43 GB of KV cache while only 3.48 GB was available. What did not fit there was the promised context length, and that is the setting to look at first.

Fixing startup errors

Set --max-model-len explicitly. Choose the longest request you actually serve, plus room for the output. In the example above, a limit of 16,384 needs 2 GiB per request and lets two run at full length at once. Newer error messages may also report an estimated maximum length that fits, which is a good starting value.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92

Raise --gpu-memory-utilization if the GPU is yours alone. Every extra percentage point goes straight to the cache. Do not go all the way to 1.0, because the process still needs headroom outside the budget.

Check what else is on the GPU. A desktop session, a notebook kernel or a second vLLM process all take memory that vLLM's fraction does not know about. If two instances share a card, their fractions have to add up to less than 1.

Use an 8-bit KV cache. --kv-cache-dtype defaults to the model's data type. Setting it to fp8, where your hardware and model support it, roughly halves the bytes per token and doubles the tokens the cache can hold. Test quality on long inputs afterwards.

Use a quantized checkpoint. A 4-bit 8B model frees around 12 GB in the example, nearly all of which goes to the cache. The docs put the cost plainly: quantized models take less memory at the cost of lower precision. We covered how to measure that in what quantization actually costs you.

Split the model. --tensor-parallel-size shards the weights across GPUs, which frees memory on each one for cache, at the cost of some synchronisation overhead.

Note that the flags are spelled with hyphens on the command line and underscores in the Python API. Mixing the two up has been a common source of "unrecognized arguments" errors.

Sizing for more than one user

Passing the startup check only proves that one request of the maximum length fits. It says nothing about concurrency. In the worked example, 4.6 GB of cache holds about 35,000 tokens in total, so the same server can run two requests of 16,384 tokens, or about eight of 4,096 tokens, before requests start waiting or being preempted.

--max-num-seqs caps how many sequences run at once, but it does not create memory. Setting it high on a small cache only means more requests competing for the same blocks. Size it from your real traffic: take typical prompt plus output length, divide the cache size by it, and use that as a starting point. vLLM's startup log reports the cache size in tokens, and on recent versions the maximum concurrency for your length as well, which saves doing the sum by hand.

Fixing warmup crashes

A raw CUDA error during warmup means vLLM set aside too much, not too little. A reported message reads: "CUDA out of memory occurred when warming up sampler with 1024 dummy requests. Please try lowering max_num_seqs or gpu_memory_utilization."

Here the direction reverses. Lower --gpu-memory-utilization a little, or lower --max-num-seqs. The memory docs also describe two ways to reduce CUDA graph memory: capture fewer batch sizes through compilation_config, or skip capture entirely with --enforce-eager. Eager mode saves memory but gives up the speed the graphs provide, so treat it as a test or a last resort.

Multimodal models can need more here too. The docs describe limit_mm_per_prompt for capping images, video or audio per prompt, and setting a modality to zero disables it.

When nothing crashes but it gets slow

If the cache fills during operation, vLLM does not fail. The optimization guide explains that when KV cache space is insufficient to handle all batched requests, requests are preempted and recomputed when space becomes available. That keeps the service alive and costs latency.

The docs suggest raising gpu_memory_utilization, lowering max_num_seqs or max_num_batched_tokens, or increasing tensor_parallel_size. Watch for this after traffic grows, or after prompts get longer, since a server sized for short prompts can start preempting without any change to the configuration.

After an upgrade

If a configuration that worked starts failing after a vLLM upgrade, it may not be your settings. The report linked above describes an upgrade from 0.7.3 to 0.8.2 in which the required GPU memory increased significantly. Defaults and memory behaviour can change between releases. Compare the startup logs from both versions, especially the reported KV cache size, before changing anything else.

A sizing checklist

1. Decide the longest request you must serve, and set --max-model-len to that.
2. Work out KV cache bytes per token from the model config: 2 × layers × KV heads × head dimension × bytes per value.
3. Multiply by your context length and your target concurrency.
4. Add the weights, and leave a margin for activations and CUDA graphs.
5. Compare the total with the GPU's memory times --gpu-memory-utilization.
6. Read the KV cache size in the startup log, which replaces your estimate with the real number.

For choosing hardware before any of this, see how much VRAM you need for a 70B model. If the bill matters as much as the fit, what it costs to run an LLM per token shows why the room left for cache decides how many requests a GPU can serve.

What we do not know

We have not measured these configurations on hardware; the worked example is arithmetic with an assumed allowance for activations. Real profiling depends on the model, the vLLM version, the attention backend and the GPU. vLLM also changes quickly, so the startup log on your machine is a better guide than any article, including this one.