"Best GPU for LLM" doesn't have one answer, because "best" depends on what you're actually doing with it — running a model locally to test a fine-tune, serving production traffic at scale, or training one from scratch. Picking the flagship card for a job that doesn't need it is the single most common way people overspend on GPU rental. Here's the honest breakdown by segment, and the one concept — memory bandwidth vs. compute — that most "best GPU" posts skip entirely.
TL;DR: For production LLM inference at scale, an H100 is the default choice — 3.35 TB/s memory bandwidth and FP8 Transformer Engine support make it the best throughput-per-dollar for serving. For local experimentation and small models, an L40S (48GB, no datacenter premium) is enough. For fine-tuning and training-heavy workloads where raw FLOPs matter more than serving latency, an A100 still holds strong cost-efficiency. The deciding factor for inference specifically is memory bandwidth, not TFLOPS — LLM token generation is bottlenecked by how fast you can stream weights out of memory, not by how many operations the chip can theoretically do.
Why memory bandwidth decides inference speed, not TFLOPS
Every "GPU comparison" post quotes TFLOPS because it's the biggest number on the spec sheet. For LLM inference, it's usually the wrong number to optimize for.
Autoregressive generation produces one token at a time. For each token, the GPU has to read the entire set of active model weights (and the KV cache) out of memory before it can do the matrix multiply that produces the next token. At small batch sizes — the common case for interactive serving — the GPU spends most of its time waiting on memory reads, not computing. This is called being memory-bandwidth-bound, and it's why a GPU with more raw FLOPS but less memory bandwidth can lose to one with fewer FLOPS and faster memory.
Compute-bound territory shows up when you batch many requests together (or during training, where you're processing full sequences of gradients, not one token at a time) — that's when TFLOPS starts to matter more than bandwidth. The practical rule: low-latency single-stream inference is bandwidth-bound; high-throughput batched serving and training are compute-bound. Most production inference workloads sit somewhere in between, which is why the GPUs with the best combination of both — H100 and H200 — dominate the "best for inference" conversation.
The spec table, from the datasheets
Every number below is from NVIDIA's own product page or datasheet.
| GPU | VRAM | Memory bandwidth | FP16 dense TFLOPS | FP8 dense TFLOPS | NVLink | Typical rental (as of Aug 2026) |
|---|---|---|---|---|---|---|
| L40S | 48GB GDDR6 | 864 GB/s | 362 TFLOPS | 733 TFLOPS | None (PCIe only) | — |
| A100 80GB SXM | 80GB HBM2e | 2.04 TB/s | 312 TFLOPS | — (no FP8 support) | 600 GB/s (3rd gen) | $1.39-2.79/hr |
| H100 SXM | 80GB HBM3 | 3.35 TB/s | 989 TFLOPS | 1,979 TFLOPS | 900 GB/s (4th gen) | $2.69-4.29/hr |
| H200 SXM | 141GB HBM3e | 4.8 TB/s | 989 TFLOPS | 1,979 TFLOPS | 900 GB/s (4th gen) | — |
Sources: NVIDIA A100 datasheet, NVIDIA H200 product page, NVIDIA L40S datasheet. Rental prices are on-demand rates checked directly from RunPod's pricing page and Lambda's GPU cloud pricing page on August 9, 2026 and May 22, 2026 respectively — check current rates before budgeting, they move.
Note the A100 row: it has no dedicated FP8 Tensor Core support — FP8 arrived with Hopper (H100/H200). If your serving stack leans on FP8 quantization for throughput, that alone rules the A100 out regardless of price.
For AMD's MI300X, we've written a dedicated inference comparison against H100/H200 — the short version is there's no single "X times faster" multiplier; it depends heavily on your serving engine and workload shape. See MI300X vs H100 vs H200: real inference benchmarks for the actual numbers instead of a vendor headline.
By what you're actually doing
Local experimentation and small-model dev
Pick: L40S (or a 24GB consumer card for anything under ~15B parameters at INT4/FP8).
You don't need datacenter-tier memory bandwidth to iterate on a fine-tune of an 8B model or test a RAG pipeline against a 30B MoE model at quantized precision. The L40S's 48GB fits most models up to 30-70B at quantized precision without the premium of an 80GB Hopper card, and its lack of NVLink doesn't matter — you're not doing multi-GPU tensor parallelism at this stage. See how much VRAM you need for LLMs for the exact sizing math per model.
High-throughput production inference
Pick: H100 (or H200 if your models or KV cache push past 80GB).
This is where the bandwidth argument matters most. The H100's 3.35 TB/s vs. the A100's 2.04 TB/s is roughly a 64% bandwidth advantage on paper, and the A100 has no FP8 Tensor Core support at all, so a model served at FP8 on H100 gets a further throughput lift the A100 architecturally cannot match. We're not going to hand you a single "H100 is Nx faster than A100" number here — real throughput depends on your model, batch size, and serving engine (vLLM, TensorRT-LLM, SGLang all extract different amounts of the hardware's ceiling), and any post that quotes one flat multiplier is quoting someone else's benchmark conditions, not yours. Benchmark your actual model and serving stack before committing to a fleet size.
If your models or context windows are large enough to make KV cache the binding constraint (long-context serving, high-concurrency chat), the H200's 141GB and 4.8 TB/s bandwidth buys you headroom the H100 doesn't have, at the same compute.
Fine-tuning
Pick: A100 for LoRA/QLoRA on a budget, H100 for full fine-tuning at scale.
Fine-tuning with LoRA or QLoRA doesn't need cutting-edge bandwidth — you're updating a small adapter, not streaming the full model repeatedly at token-by-token latency targets. The A100's lower rental price (roughly half of H100 on most platforms, per the table above) makes it the better cost-per-training-run choice for LoRA workflows on models up to 70B. Full fine-tuning, which needs to hold weights + gradients + optimizer state simultaneously (see our VRAM sizing breakdown for the exact math — it's roughly 9x the inference weight footprint), benefits more from H100's throughput because you're compute-bound on full backward passes across the whole model, not memory-bound on generation.
Full pretraining
Pick: H100 or H200 clusters with NVLink, not a decision most teams need to make.
If you're pretraining a model from scratch, you're deep enough into this that "best GPU" is really a networking and cluster-topology question (NVLink/NVSwitch bandwidth between GPUs matters as much as any single card's spec), and you're almost certainly working with a provider on dedicated capacity rather than picking a GPU off a marketplace listing. Worth naming so this list is honest: this segment exists, and it's a different problem than the rest of this post.
What "best" actually costs
Sticker price per hour is not cost per completed job. An H100 at roughly 1.5-2x the A100's hourly rate that finishes an inference or fine-tuning job in half the time is cheaper in total, not more expensive. Conversely, renting an H100 to run an 8B model at low concurrency is paying for bandwidth and FP8 throughput you'll never touch. Match the segment above to your actual workload, then check live pricing across providers on the GPU availability index or run your numbers through the GPU recommender — and see current rates for every tier on our pricing page.
Frequently asked questions
What is the best GPU for LLM inference?
For production-scale serving, the H100 is the best combination of memory bandwidth (3.35 TB/s) and FP8 compute support, both of which directly determine token generation speed. For local development or smaller models, a 48GB L40S is enough and considerably cheaper.
Is the A100 or H100 better for fine-tuning?
For LoRA/QLoRA fine-tuning, the A100 is usually the better cost-per-run choice since adapter training doesn't need Hopper's bandwidth or FP8 support, and A100 rents for roughly half the price. For full fine-tuning at scale, H100's higher compute throughput pays for itself on large backward passes.
Why does memory bandwidth matter more than TFLOPS for LLM inference?
Because autoregressive token generation reads the full set of active weights and KV cache from memory for every single token, at low batch sizes the GPU spends more time waiting on memory than computing — this is called being memory-bandwidth-bound. TFLOPS matters more once you're batching heavily or training, where you're compute-bound instead.
What's the best GPU for fine-tuning on a budget?
An A100 80GB running LoRA or QLoRA is the standard budget choice — it fits most models up to 70B at quantized precision for the adapter training, and its hourly rate is roughly half of an H100's across most rental platforms as of August 2026.
Does the MI300X beat NVIDIA GPUs for LLM inference?
There's no single multiplier that holds across workloads — independent benchmarks show it's competitive with H200 at some latency targets on large models and behind at others, depending heavily on the serving engine used. See our dedicated MI300X vs H100/H200 comparison for the actual benchmark data.
Whichever tier fits your workload, the GPU you pick shouldn't lock you to one provider — pause and resume means you can move a running environment between H100, A100, or L40S nodes across providers as your workload's needs change, without rebuilding from scratch each time.
