If you're looking for a single number — "the MI300X is 1.8x faster than an H100" — every credible source I could find says that number doesn't exist. Real MI300X-vs-H100 inference results are workload-, batch-, and latency-target-dependent, and they flip direction depending on what you measure. This post pulls the actual benchmark results I could verify, names the ones I couldn't, and explains the software gap that decides more than the memory-bandwidth spec sheet does.
TL;DR: No single scalar comparison holds up. The clearest independent data point (SemiAnalysis, May 2025) found MI300X/MI325X competitive with or ahead of H200 at some latency targets on large models, and behind at others — it depends on the target. vLLM's own benchmark shows serving-engine choice matters as much as hardware: 1.5-1.8x throughput gains just from picking vLLM over TGI on the same MI300X box. The honest read: the MI300X's memory advantage is real, but ROCm's kernel maturity is still catching up to CUDA's, and that gap — not the spec sheet — is what decides your actual tokens/sec today.
Why there's no clean multiplier
Inference throughput is a function of model size, batch size, sequence length, precision, the serving engine, and the specific kernel path that engine takes on that hardware — not of TFLOPS alone. Two boxes with identical peak FLOPS can differ by 2x on real workloads because one has a better-tuned attention kernel for that exact model shape. That's true on NVIDIA too, but ROCm has fewer years of kernel tuning behind it, so the variance is larger and moves faster from one software release to the next.
That's the reason vendor comparisons disagree so much: whichever one you read picked the batch size, sequence length, and software version that made their hardware look best. None of that makes the numbers below wrong — it means you should read what conditions produced them, not just the headline ratio.
What the independent benchmark found
The most careful third-party comparison I could verify is SemiAnalysis's AMD vs NVIDIA inference benchmark (May 23, 2025), which explicitly discloses support relationships with both AMD and NVIDIA rather than being sponsored by either. It tested vLLM, SGLang, and TensorRT-LLM at FP8 and FP16 on Llama 3 405B and DeepSeek V3 (670B) — models large enough that MI300X's memory advantage should matter.
The result was not a flat multiplier. MI300X and MI325X were competitive with or ahead of H200 at some latency targets on the largest models, and H200 won at low-to-medium latency targets on DeepSeek. Which chip wins depends on where your latency SLA sits on the curve, not on a single "faster/slower" verdict. If someone hands you one number from this benchmark without the latency target attached, ask which point on the curve it's from.
What vLLM's own numbers show — and what they actually measure
vLLM's own blog post on AMD serving (October 23, 2024, vLLM 0.6.2) published real throughput numbers on 8x MI300X:
| Model | vLLM throughput | TGI throughput | vLLM TTFT advantage |
|---|---|---|---|
| Llama 3.1 405B, BF16, 32 QPS | 5.76 req/s | 3.55 req/s | 1.7x faster |
| Llama 3.1 70B | ~1.8x throughput | baseline | 5.1x faster |
Read this carefully: it's vLLM vs TGI on the same MI300X hardware, not MI300X vs H100. It's real, and it's useful — it shows that on AMD, the serving engine you pick moves throughput as much as a hardware generation would. But it doesn't answer the vs-NVIDIA question at all, and I'd be misrepresenting it if I implied otherwise.
MLPerf — submitted, but no clean head-to-head number
AMD has submitted MI300X to MLPerf Inference (v4.1, August 2024) and later rounds, including v5.1 (September 2025) alongside H100/H200/B200 submissions. MLCommons' results live behind an interactive dashboard rather than a static comparison table, and I wasn't able to pull a clean Llama2-70B MI300X-vs-H100 scalar out of it for this post. If you need that specific number, go to mlcommons.org's inference benchmark results and filter it yourself — treat any blog post (including this one) that quotes an MLPerf multiplier without a direct link to the results row with skepticism.
I also found AMD's own 2023 launch-day comparison claims (a 1.4x-to-2.1x MI300X-vs-H100 figure circulates widely) but could not independently re-verify the exact conditions behind that number from AMD's own page at the time of writing, so I'm not repeating it as fact here. If you find it, check the batch size and precision it was measured at before you trust it.
The real gap: kernel maturity, not silicon
The more useful signal for "what will actually happen when I deploy" isn't a benchmark, it's the state of open issues against the exact software you'll run:
- vllm-project/vllm#19692 (June 2025) — vLLM's V1 engine runs slower than the older V0 engine on MI300X for Llama 3 8B, the opposite of the regression direction seen on NVIDIA, suspected to trace to the Triton attention kernel.
- vllm-project/vllm#28052 (November 2025) — the flash-attention backend on
rocm/vllm:rocm7.0.0throws a HIP device-function error on Qwen3-VL-8B, a regression between vLLM releases. - vllm-project/vllm#50064 (July 2026) — ROCm's sparse-MLA metadata is missing a field CUDA's backend has, blocking startup for sparse-MLA models like DeepSeek on MI300X.
None of these are fatal — they're the normal texture of a newer software stack, and vLLM's own ROCm attention-backend post (February 2026, vLLM 0.14 / ROCm 7.0) shows the pace of improvement: switching the attention backend from ROCM_ATTN to ROCM_AITER_FA alone gave 3.6-4.4x throughput at 64 concurrent requests across the MI300X/MI325X/MI355X line, and an MLA kernel swap gave 1.3-1.5x on DeepSeek-R1. That's a bigger jump than any hardware generation delivers, from a config flag. Which means the honest answer to "is MI300X fast enough for my model" changes month to month as these land — pin your ROCm and vLLM/SGLang versions and re-test rather than trusting a benchmark from six months ago.
What this means if you're deciding today
Pick MI300X when your model is memory-bound (long context, large batch, a 70B+ model that needs the 192GB to avoid tensor-parallel splitting) and you're running a mainstream serving stack — vLLM or SGLang, both of which get regular ROCm-specific kernel work. The SemiAnalysis data says you're competitive on cost-per-token at large model sizes, not automatically ahead.
Be cautious when you're pinned to a specific serving engine version, need a kernel path that isn't AITER/composable-kernel-optimized yet, or your model architecture is new enough that ROCm support might trail CUDA by a release or two (sparse-MLA above is a live example). Test your exact model + engine + precision combination before committing — by-the-minute billing exists for exactly this.
We wrote up the hardware spec case and current MI300X pricing separately in renting an MI300X; this post is the inference-performance half of that decision. For the ROCm software-compatibility side specifically — what runs and what doesn't for image generation — see our ComfyUI and PyTorch on AMD ROCm writeup.
FAQ
Is the MI300X faster than the H100 for inference? There's no single answer. Independent benchmarks show it's competitive with or ahead of H200 (a newer, larger-memory chip than H100) at some latency targets on large models, and behind at others. It depends on your specific model, batch size, and latency target more than on the chip alone.
What serving engine should I use on MI300X? vLLM and SGLang both receive active ROCm-specific kernel work. vLLM's own benchmark showed 1.5-1.8x throughput over TGI on the same MI300X hardware — the engine choice matters as much as anything else in this comparison.
Has AMD submitted MI300X to MLPerf? Yes, starting with MLPerf Inference v4.1 (August 2024) through at least v5.1 (September 2025). The results live in MLCommons' interactive dashboard rather than a simple table.
Why do MI300X benchmarks disagree so much between sources? Because throughput depends on batch size, sequence length, precision, and serving-engine software version, and ROCm's kernel maturity is improving fast enough that a benchmark from even a few months ago can be stale. A single config-flag change (switching attention backends) produced a 3-4x throughput jump in one vLLM release.
Sources
- SemiAnalysis: AMD vs NVIDIA inference benchmark, May 23, 2025. vLLM/SGLang/TensorRT-LLM at FP8/FP16, Llama 3 405B and DeepSeek V3.
- vLLM: Serving LLMs on AMD MI300X, October 23, 2024, vLLM 0.6.2. Llama 3.1 405B/70B, vLLM vs TGI.
- MLPerf Inference v4.1 results, August 2024. MLPerf Inference v5.1 results, September 2025. MLCommons inference results dashboard.
- vllm-project/vllm#19692, #28052, #50064 — open/recent ROCm-specific issues, checked August 20, 2026.
- vLLM ROCm attention backend post, February 27, 2026, vLLM 0.14.0rc2 / ROCm 7.0.0.
- SGLang releases — v0.5.14-0.5.17, 2026, MI300X/MI325X/MI355X support notes.
