LLM VRAM Calculator: Which GPU Do I Need?

The formula

GPU memory for an LLM has four parts:

Part	Rule of thumb
Weights	parameters × bytes per parameter: 4 (FP32), 2 (BF16/FP16), 1 (FP8/INT8), 0.5 (INT4)
KV cache	2 × layers × context × batch × KV width × 2 bytes
Training state	full fine-tune with Adam in mixed precision is about 16 bytes per parameter before activations (Hugging Face)
Overhead	CUDA context, fragmentation and framework buffers; we add 10%

Quick answers

Model and task	Approximate VRAM	Smallest common fit
8B inference, BF16, 8k context	about 19 GB	one 24 GB card (L4, RTX 4090)
70B inference, BF16, 8k context	about 158 GB	1 × B200, 2 × H200 or 4 × H100
70B inference, INT4	about 42 GB	one 48 GB L40S
70B QLoRA fine-tune	about 50–60 GB	one A100 or H100 80 GB
70B full fine-tune	over 1 TB	multiple 8-GPU nodes

These are estimates. Long contexts and large batches change the KV cache a lot, so set yours above.

Why the KV width matters

Modern models share key and value heads (grouped-query attention). Llama 3 70B has 64 query heads but only 8 KV heads of size 128, so its KV width is 1,024, not 8,192. That cuts the cache by 8 times compared with older models.