What GPU do I need to run meta-llama/Llama-2-7b-chat-hf?
6.7B parameters, published in F16. View on Hugging FaceGated
VRAM required & cheapest live GPU fit
Required VRAM = weight size at each precision × 1.2, a documented allowance for KV-cache, activations, and allocator fragmentation at an assumed context length of 8,192 tokens. This is a rule-of-thumb, not a simulation — long contexts or large batch sizes need materially more headroom.
“Cheapest live fit” only considers GPUs whose hardware actually supports the row's precision (e.g. a pre-Ampere card is never matched to a BF16 or FP8 row, regardless of its VRAM or price) and caps a fit at 8 GPUs of the same model. When a single GPU fits, it's shown as the primary recommendation even if a cheaper multi-GPU option of older cards exists — that cheaper option is listed as a secondary “cheaper alt.” row instead.
INT4 caveat: the INT4 row requires BOTH a GPU with AWQ/GPTQ/Marlin-class quantized-serving kernel support (Turing or newer — Pascal/Volta/older cards never qualify, same as this table already enforces) AND a quantized checkpoint that's actually been published for this model, which is not guaranteed and isn't verified here. Check the model's Hugging Face page for a published INT4/AWQ/GPTQ variant before relying on this row.
Cheapest way to run Llama-2-7b-chat-hf at its published (F16) precision: 1× V100 on simplepod, at $0.070/hr per GPU ($0.070/hr total). Quantizing to FP8 or INT4 (rows above) can cost less, but requires a compatible quantized checkpoint to exist for this model.
Assumptions
- Weight bytes = parameter count × bytes-per-parameter for the shown precision (FP32=4, BF16/FP16=2, FP8=1, INT4=0.5 bytes).
- Required VRAM adds a flat 1.2× overhead for KV-cache, activations, and fragmentation, calibrated for an assumed 8,192-token context — not a per-request simulation.
- A GPU is only matched to a row if its hardware actually supports that row's compute format — pre-Ampere cards (Pascal/Volta/Turing) never qualify for a BF16 or FP8 row, regardless of VRAM or price. See the GPU Index for per-model specs.
- FP8/INT4 rows assume clean post-training quantization is available for this architecture; actual quality loss and tooling support vary by model.
- Prices are live per-GPU $/hr rates from Aquanode's marketplace feed, normalized via the same methodology as the GPU Index, with outlier offers excluded.
- Multi-GPU fits assume the model shards cleanly across cards (e.g. tensor parallelism) — real throughput and interconnect overhead aren't modeled here.
