How Much VRAM Do You Need for a 70B Model?

A 70B model needs about 140 GB of VRAM at 16-bit, 70 GB at 8-bit and 35 to 40 GB at 4-bit, for the weights alone. On top of that comes memory for context, which is about 2.5 GiB per 8,000 tokens for a Llama 3 70B style model. So a realistic minimum for a 4-bit 70B model at moderate context is 48 GB, which means two 24 GB cards or one 48 GB card.

Those numbers are worth being able to work out yourself, because the answer changes with the model, the precision and how long your prompts are. It takes two small sums.

Step 1: the weights

Weights take the number of parameters times the bytes per parameter. With parameters in billions, the answer comes out in gigabytes:

VRAM for weights in GB = parameters in billions × bytes per weight

Model size	16-bit (2 bytes)	8-bit (1 byte)	4-bit (about 0.5 bytes)
8B	16 GB	8 GB	4 to 5 GB
27B	54 GB	27 GB	14 to 16 GB
32B	64 GB	32 GB	16 to 19 GB
70B	140 GB	70 GB	35 to 40 GB

The 4-bit column is a range because real files are larger than the simple sum. They store scale factors, and some layers are usually kept at higher precision. For a 70B model, published estimates of practical 4-bit size tend to sit around 40 GB. The size of the file you download is the best number to use. If it is 42 GB on disk, it will take at least 42 GB of VRAM.

Quantization has a quality cost as well as a memory saving, which we covered in what quantization actually costs you.

Step 2: the context

Every token in a conversation stores keys and values in the KV cache, and this is the part most VRAM guides leave out. Per token:

KV cache bytes per token = 2 × layers × KV heads × head dimension × bytes per value

All of these are in the model's config.json. Llama 3 70B's configuration lists 80 layers, 8 key-value heads, and a hidden size of 8,192 across 64 attention heads, which gives a head dimension of 128. At 16-bit:

2 × 80 × 8 × 128 × 2 = 327,680 bytes, or 320 KiB per token

Here is what that means for three common shapes of model, at 16-bit, for a single request:

Model shape	Per token	8K tokens	32K tokens	128K tokens
8B (32 layers, 8 KV heads)	128 KiB	1 GiB	4 GiB	16 GiB
32B (64 layers, 8 KV heads)	256 KiB	2 GiB	8 GiB	32 GiB
70B (80 layers, 8 KV heads)	320 KiB	2.5 GiB	10 GiB	40 GiB

Two things stand out. At 128,000 tokens, the cache of a 70B model is larger than its 4-bit weights. And these figures are per request, so four people using a long context at once need four times as much. That is why the KV cache decides how many requests a GPU can serve.

Storing the cache at 8 bits halves those numbers, at some cost in quality that you should test on long inputs.

Step 3: leave a margin

The runtime needs memory for activations, its own buffers and, on some engines, CUDA graphs. How much varies by engine and model, so we would budget a margin of 10 percent or a few GB and then check the real figure. Serving engines also reserve memory up front, so a tool like nvidia-smi showing the card nearly full does not mean you are out of room.

Putting it together:

Total VRAM ≈ weights + (KV cache per token × tokens × concurrent requests) + margin

What actually fits a 70B model

Here is the sum for a 70B model and one request, using the figures above. It is arithmetic, not a measurement.

Precision	Weights	Context	KV cache	Total with margin	Fits in
4-bit	≈ 40 GB	8K	2.5 GiB	≈ 46 GB	48 GB
4-bit	≈ 40 GB	32K	10 GiB	≈ 55 GB	64 GB or more
8-bit	70 GB	8K	2.5 GiB	≈ 78 GB	80 GB, tightly
16-bit	140 GB	8K	2.5 GiB	≈ 150 GB	160 GB, across GPUs

The first two rows show how quickly context changes the answer: the same 4-bit model moves out of a 48 GB budget.

Total VRAM	What to expect with a 70B model
24 GB	Does not fit. Needs offloading, and it will be slow
32 GB	Does not fit a typical 4-bit file
48 GB	4-bit at moderate context
80 GB	4-bit with long context or several users; 8-bit with short context
160 GB	16-bit, or 8-bit with plenty of room for users

A second example: a 27B model on a 24 GB card

The same method works for any size. Suppose you want a 27B model on one 24 GB card, the most common local setup. At 4-bit the weights take 14 to 16 GB, which leaves about 8 GB. After a margin of 2 GB, about 6 GB remains for context. The exact cache size depends on that model's config.json, but at 256 KiB per token, the figure for the 32B shape above, 6 GB holds a little over 22,000 tokens. That is comfortable for one person and tight for several. At 8-bit the weights alone take 27 GB, so that option is out on this card.

Using two GPUs

VRAM across cards adds up for this purpose, as long as the engine splits the model. vLLM calls this tensor parallelism and describes it as splitting the model across multiple GPUs, with the tensor_parallel_size option. Two 24 GB cards give you about 48 GB, less a little overhead on each card. Communication between the cards costs some speed, and the link between them matters.

Memory on two cards is not the same as memory on one, though. If the model does not divide evenly, one card fills first, and that card sets the limit.

What happens if it does not fit

Most local runners will offload what does not fit to system memory. The model then runs, but layers held in system RAM have to move across the PCIe bus or be computed on the CPU, and generation can drop to a small fraction of its normal speed. vLLM has --cpu-offload-gb for this. It is fine for a test and rarely fine for daily use.

Serving engines are stricter. vLLM will refuse to start if the context you asked for does not fit, which we explain in why vLLM runs out of memory. Usually the fix there is to ask for a shorter context, not to buy a bigger GPU.

Fine-tuning needs more

Everything above is for inference. Full fine-tuning needs memory for gradients and optimizer state as well, which takes it far beyond these numbers. QLoRA is the usual way around that. Its authors report fine-tuning a 65 billion parameter model on a single 48 GB GPU while preserving 16-bit fine-tuning task performance, by training small adapters on top of a frozen 4-bit model.

Do you need a 70B model at all?

Before buying hardware, it is worth asking. A 32B model at 4-bit fits a single 24 GB card with room for context, and on narrow, repeatable tasks small models often do the job. A mixture-of-experts model goes the other way: it computes like a small model but needs the memory of a large one, so size it by total parameters, not active ones. Test candidates on your own task, since a leaderboard will not tell you which model to run.

A quick way to size anything

1. Find the file size of the model at the precision you want. That is your weights figure.
2. Open config.json and work out KV cache per token.
3. Multiply by the longest context you need and the number of people using it at once.
4. Add a margin of about 10 percent.
5. Compare with your total VRAM, and if it is close, shorten the context first.

What we do not know

These are calculations, and we have not measured them on hardware. Real use depends on the engine, the quantization format, and the model; some newer architectures reduce the KV cache in ways this formula does not capture, so read the model's own documentation. Treat the tables as a first estimate, then load the model and read what it actually uses.