GPU benchmarks for deep learning

The TFLOPS and bandwidth figures above are the ceiling: the most a chip can theoretically move or compute per second, published by the vendor under ideal conditions. No real training or inference job hits that ceiling, because a real job is bottlenecked by whichever resource runs out first, and which one that is changes with the workload.

Training a model from scratch is usually compute-bound: the GPU spends most of its time doing matrix multiplication, so a card with a higher dense TFLOPS figure (like an H200 or B200 over an A100) trains faster on the same batch. Serving an LLM is often memory-bandwidth-bound instead: generating each token means streaming the entire model's weights out of memory, so bandwidth, not TFLOPS, decides throughput once the model already fits in VRAM. That is why H200 (4.8 TB/s) can out-serve an H100 (3.35 TB/s) on a long-context workload even though the two chips have identical compute.

Precision matters too. FP8 roughly doubles the throughput of FP16 on the same chip, which is why newer serving stacks quantize to FP8 wherever accuracy allows it, and why a spec sheet published in FP8 is not directly comparable to one published in FP16.

MLPerf is the industry-standard, independently audited benchmark for training and inference throughput, run by MLCommons with submissions from the hardware vendors and cloud providers themselves. A few results from recent rounds, with the caveats that make each one honest rather than a marketing headline:

• Fine-tuning: on the Llama 2 70B LoRA fine-tuning benchmark in MLPerf Training v4.1, NVIDIA reported that "a single HGX B200 server delivers 2.2X more performance compared to an HGX H100 server", comparing an 8-GPU DGX B200 submission against an 8-GPU HGX H100 submission (NVIDIA developer blog).
• Pretraining: in the same round, NVIDIA reported Blackwell delivering roughly twice the per-GPU performance of Hopper on GPT-3 175B pretraining, normalizing a 64-GPU B200 submission against a 256-GPU H100 submission (NVIDIA developer blog).
• Inference: in MLPerf Inference v5.0, CoreWeave's 8x H200 node reached 33,000 tokens/second on the Llama 2 70B offline scenario at FP8, which CoreWeave describes as 40% higher throughput than the fastest H100 submission in the prior (v4.1) round at FP16, a real gain, though the two submissions run at different precisions, not an apples-to-apples rerun on identical settings (CoreWeave).
• Frontier scale: on the newer, larger Llama 3.1 405B inference benchmark, CoreWeave's GB200 NVL72 instances reached over 800 tokens/second at FP4, a 2.86x per-chip speedup over H200 on the same benchmark (CoreWeave).

None of these numbers are Aquanode's own measurements. They are what the submitting organizations published to MLCommons, and every one of them is a specific system configuration at a specific precision, not a universal multiplier that applies to every workload. Run the full, current results yourself at mlcommons.org.