You want to run a model and you don't know if your GPU fits it. Most answers to "how much VRAM do I need" are a vibe — "70B needs like 40GB probably." That's wrong often enough to waste an afternoon on an out-of-memory crash halfway through loading weights. There's an actual formula underneath this, it takes three lines to compute, and once you have it you can size any model against any GPU in your head.
TL;DR: VRAM needed ≈ (parameters × bytes-per-parameter) + KV cache + ~10-20% overhead. A 70B model at FP16 (2 bytes/param) needs ~140GB for weights alone — too big for one 80GB GPU — but at 4-bit quantization (0.5 bytes/param) it drops to ~35-40GB and fits on a single A100 80GB or two RTX 4090s. An 8B model runs comfortably on a single 24GB card at FP16. The KV cache adds more on top of weights as your context length and batch size grow, and full fine-tuning needs roughly 8x the weight memory just for the optimizer, which is why almost nobody full-fine-tunes anymore.
The three numbers that decide everything
VRAM usage for running a model breaks into three pieces, and only one of them is fixed by the model itself.
1. Model weights. This is params × bytes-per-parameter:
| Precision | Bytes/param | 8B model | 70B model |
|---|---|---|---|
| FP16 / BF16 | 2 | 16 GB | 140 GB |
| FP8 | 1 | 8 GB | 70 GB |
| INT4 / Q4 (quantized) | 0.5 | 4 GB | 35 GB |
This is the standard convention used by Hugging Face's own model-memory documentation and it's the number most people stop at — which is why estimates run low.
2. KV cache. Every token you generate gets its key and value vectors cached so the model doesn't recompute attention over the whole sequence each step. NVIDIA's inference optimization guide gives the formula:
KV cache (bytes) = 2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × bytes_per_param
The 2 accounts for storing both keys and values. num_kv_heads matters a lot here — modern models use grouped-query attention (GQA), where far fewer KV heads exist than attention heads, which is the single biggest reason KV cache is smaller than people assume from older architectures.
Concretely, for Llama 3.1 8B (32 layers, 8 KV heads, head_dim 128) at FP16, batch size 1: a 4,096-token context costs 2 × 32 × 8 × 128 × 4096 × 2 bytes ≈ 537 MB. At 128K context (its full window), that scales to roughly 17 GB — bigger than the model's own weights. Context length is not free, and it's the part most VRAM estimates skip entirely.
3. Overhead. Activations, CUDA kernels, framework buffers (PyTorch, vLLM, etc.) typically add 10-20% on top of weights + KV cache. Budget for it; don't assume you can fill a GPU to the last gigabyte.
Model VRAM table: what fits where
Every parameter count below is from the model's own Hugging Face card or official release post. Every GPU VRAM figure is from the vendor's own datasheet.
| Model | Params (total / active) | FP16 weights | FP8 weights | INT4 weights | Fits on |
|---|---|---|---|---|---|
| Llama 3.1 8B | 8B dense | 16 GB | 8 GB | 4 GB | Any 24GB card (RTX 4090) at any precision |
| Mistral 7B v0.3 | 7.3B dense | ~15 GB | ~7 GB | ~4 GB | Any 24GB card |
| Qwen3-30B-A3B | 30.5B / 3.3B active | ~61 GB | ~31 GB | ~15 GB | L40S 48GB at FP8; single 24GB card at INT4 |
| Llama 3.1 70B | 70B dense | 140 GB | 70 GB | 35 GB | A100 80GB at FP8/INT4; 2× 80GB GPUs at FP16 |
| Qwen3-32B | 32.8B dense | ~66 GB | ~33 GB | ~16 GB | Single A100/H100 80GB at FP16; one 24GB card at INT4 |
| Qwen3-235B-A22B | 235B / 22B active | ~470 GB | ~235 GB | ~118 GB | H200 141GB at INT4; multi-GPU at FP8+ |
| Llama 3.1 405B | 405B dense | ~810 GB | ~405 GB | ~203 GB | Multi-GPU H100/H200 node at FP8; ~3× H200 at INT4 |
| DeepSeek-V3 | 671B / 37B active | ~1,342 GB | ~671 GB | ~336 GB | Multi-GPU node (MI300X 192GB ×2+ at FP8) |
A few things worth being precise about. MoE models are cheap to run and expensive to hold. Qwen3-30B-A3B only computes like a 3.3B model per token — that's why it's fast — but every one of its 30.5B parameters still has to sit in VRAM, because you don't know in advance which experts a given token will route to. Same logic applies to DeepSeek-V3's 671B total against 37B active: the active-parameter number tells you about latency and throughput, not about whether the model fits in memory.
Quantization error is real, not free. INT4 weights are the smallest number in the table, but aggressive quantization measurably degrades output quality, especially on reasoning-heavy tasks — treat the INT4 column as "what's technically loadable," and validate the output quality on your actual task before committing to it in production.
Training and fine-tuning need a different formula entirely
Almost every "how much VRAM" search is really a training question in disguise, and training's memory footprint is not the inference formula scaled up — it's a different calculation.
Full fine-tuning with mixed-precision AdamW needs roughly 18 bytes per parameter, per Hugging Face's own breakdown: 6 bytes for two copies of the weights (FP16 for the forward/backward pass, FP32 "master" copy for stable updates), 4 bytes for FP32 gradients, and 8 bytes for Adam's two FP32 moment buffers (momentum + variance) — 18 bytes/param before activations are even counted. That means full-fine-tuning an 8B model needs roughly 144GB just for weights, gradients and optimizer state — before a single activation is stored. This is why full fine-tuning of anything past ~7B is a multi-GPU exercise for almost everyone.
LoRA and QLoRA sidestep this by freezing the base weights and only training small low-rank adapter matrices, so you pay the inference memory cost for the frozen base (optionally quantized to INT4 with QLoRA) plus a small amount for the trainable adapters and their optimizer state — typically single-digit percent of full fine-tuning's footprint. This is why QLoRA can fine-tune a 70B model on a single 48GB L40S or 80GB A100, where full fine-tuning of the same model would need well over 1TB of VRAM.
| Training method | Approx. memory vs. inference weights | Typical hardware for 70B |
|---|---|---|
| Full fine-tuning (Adam, mixed precision) | ~9x FP16 weight size | Multi-GPU H100/H200 node |
| LoRA | Base model + small adapter overhead | Single 80GB GPU |
| QLoRA (4-bit base) | ~INT4 weight size + adapter overhead | Single 48GB or 80GB GPU |
Doing the math yourself
You don't need to memorize the table above for every model you'll ever touch. Use the GPU recommender to plug in a model and quantization and get a hardware match, or work it by hand:
- Look up the model's parameter count on its Hugging Face card.
- Multiply by bytes-per-parameter for your target precision.
- Add KV cache for your expected context length and batch size (bigger for long-context or high-concurrency serving, negligible for short single-user chat).
- Add 15% overhead.
- Check the result against the GPU availability index for what fits at what price, across providers.
If the number is close to a GPU's exact VRAM figure, don't cut it that fine — leave headroom, because framework overhead and any batch-size spike will push you over.
Frequently asked questions
How much VRAM do I need to run Llama 3.1 70B?
At FP16, ~140GB for weights alone, which doesn't fit on a single 80GB GPU. At FP8 it drops to ~70GB (fits one A100/H100 80GB with headroom for a modest KV cache), and at INT4 it drops to ~35GB, fitting comfortably on one 80GB card or even a 48GB L40S with a small context window.
What GPU do I need to run a local LLM?
For 7-8B models, any 24GB consumer or datacenter card (RTX 4090, L40S) handles FP16 comfortably. For 30-70B models, you need either an 80GB datacenter GPU (A100, H100) or quantization down to INT4 to fit a smaller card. Above 100B parameters, plan for multi-GPU regardless of quantization.
How is VRAM different for training vs. inference?
Inference only needs to hold the weights plus a KV cache. Training needs the weights, gradients, and optimizer state simultaneously — full fine-tuning with Adam needs roughly 18 bytes per parameter versus 2 bytes for FP16 inference, close to a 9x jump. LoRA/QLoRA avoid most of that by freezing the base model.
Does quantization hurt model quality?
Yes, to some degree — INT4 quantization trades accuracy for memory and speed, and the size of the hit depends on the model and task. Reasoning-heavy tasks tend to show it more than straightforward text generation. Test your specific task's output quality before deploying a quantized model to production, don't assume the benchmark numbers transfer.
Why do MoE models like DeepSeek-V3 need so much VRAM if they only "activate" a fraction of parameters?
Because you don't know in advance which experts a given token will route to, so every expert's weights have to be resident in VRAM even though only a subset computes per token. The active-parameter count predicts speed and compute cost, not memory footprint — the total parameter count is what you size VRAM against.
Once you know what fits, the next question is which GPU is worth renting for it — see our companion post on choosing the best GPU for LLM inference. And if you're moving between quantization levels or GPU generations mid-project, pause and resume means you don't lose your environment switching providers to find one that has the VRAM you need in stock.
