Every "TPU vs GPU" search eventually lands on a page that says "TPUs are built for AI, GPUs are more general purpose" and stops there. That's true and useless — it doesn't tell you whether to pin your next training run to Google Cloud or rent an H100 somewhere you already have infrastructure. This post goes down to the actual hardware and software difference, with current specs from NVIDIA and Google's own pages, and ends with a framework you can apply to your workload today.
TL;DR: A TPU is a systolic-array chip built to run one thing — dense matrix multiplication compiled ahead of time by XLA — extremely efficiently; a GPU is a general-purpose SIMT (single-instruction-multiple-thread) processor that runs matrix math well but can also run arbitrary parallel code. Google's current-generation TPU (Ironwood / TPU7x) packs 192GB HBM at 7.37 TB/s per chip and scales to 9,216 chips per pod; NVIDIA's H100 SXM packs 80GB HBM3 at 3.35 TB/s per GPU and scales via NVLink/InfiniBand into arbitrary cluster sizes. TPUs are Google Cloud-only and JAX-first; GPUs run everywhere and own the CUDA/PyTorch ecosystem. Pick a TPU for large, standard transformer training/inference at Google Cloud scale where you can live inside JAX; pick a GPU for everything that needs ecosystem flexibility, multi-cloud portability, or a framework/kernel that isn't XLA-compiled yet.
What a TPU actually is
A Tensor Processing Unit is Google's custom ASIC (application-specific integrated circuit), built around a systolic array: a grid of multiply-accumulate units that pass partial results directly to their neighbors, clocked in lockstep, instead of reading and writing back to memory between every operation. For a workload that's mostly large matrix multiplications — which transformer training and inference are — this removes most of the memory traffic a general-purpose processor would spend moving intermediate values in and out of registers and caches. It's why a TPU's compute-per-watt on dense matmul is high: the chip does almost nothing else.
The tradeoff is the programming model. TPUs are programmed through XLA (Accelerated Linear Algebra), a compiler that traces your whole computation graph ahead of time and compiles it into a fixed, static execution plan for the systolic array. That's fast once compiled, but it means the TPU is fundamentally a compile-then-run device: dynamic control flow, data-dependent shapes, and custom low-level kernels are harder to express than on a GPU, because XLA needs to know the shape and structure of the computation before it runs.
What a GPU actually is
A GPU uses the SIMT model: thousands of small cores execute the same instruction stream across many independent threads, with each thread free to branch, index dynamically, and read/write memory directly. NVIDIA's H100, for instance, has 132 enabled streaming multiprocessors (SMs) on the SXM5 variant, each capable of scheduling and running its own warps of threads. Layered on top of the general SIMT cores are Tensor Cores — dedicated matrix-multiply units, the part of the chip that actually does the FP8/FP16/BF16 tensor math that dominates AI workloads — plus dedicated Transformer Engine logic on Hopper and Blackwell for FP8 precision management.
The result: a GPU can run a matmul-heavy transformer at high throughput via its Tensor Cores, and also run arbitrary CUDA kernels, video decode, physics simulation, or a training loop with dynamic control flow, in the same process. It's a slightly less efficient matmul engine per watt than a systolic array, in exchange for being a general-purpose parallel processor.
Current generation, spec for spec
Google's current TPU generation available on Cloud is Ironwood (TPU7x), its seventh generation, alongside the still-available Trillium (TPU v6e). NVIDIA's current data-center lineup spans H100 (Hopper, in volume production since 2023 and still widely available to rent), H200 (Hopper refresh with more/faster memory), and B200 (Blackwell, the newest generation).
| TPU v6e (Trillium) | TPU7x (Ironwood) | H100 SXM5 | H200 SXM | |
|---|---|---|---|---|
| Peak compute/chip (BF16) | 918 TFLOPS | 2,307 TFLOPS | 989 TFLOPS (dense) | 989 TFLOPS (dense) |
| Peak compute/chip (FP8/INT8) | 1,836 TOPS | 4,614 TFLOPS | 1,979 TFLOPS (dense) | 1,979 TFLOPS (dense) |
| Memory | 32 GB HBM | 192 GB HBM | 80 GB HBM3 | 141 GB HBM3e |
| Memory bandwidth | 1,638 GB/s | 7,370 GB/s | 3.35 TB/s | 4.8 TB/s |
| Chip-to-chip interconnect | 800 GB/s (ICI) | 1,200 GB/s (ICI) | 900 GB/s (NVLink) | 900 GB/s (NVLink) |
| Max scale | 256 chips/pod | 9,216 chips/pod | Cluster-scale via NVLink+InfiniBand | Cluster-scale via NVLink+InfiniBand |
Two things stand out reading this table straight. First, Ironwood's per-chip memory bandwidth (7.37 TB/s) is more than double an H200's, because Google is optimizing for feeding a systolic array that never stalls waiting on data. Second, NVIDIA's numbers describe a single, sellable GPU you can rent from dozens of providers in any hourly increment; Google's numbers describe chips that are provisioned as TPU "slices" (pods), a fundamentally different purchasing unit. That difference matters more than the raw TFLOPS once you get to availability and pricing.
For context on Blackwell: NVIDIA's HGX B200 8-GPU system datasheet lists 1.4 TB total HBM3e across 8 GPUs (roughly 175 GB per GPU) and 144 PFLOPS FP4 Tensor Core (sparse) system-wide, on fifth-generation NVLink at 1.8 TB/s GPU-to-GPU — a further jump on both memory and raw throughput over H100/H200, at the cost of a new precision format (FP4) that not every framework and kernel supports yet.
The software reality: XLA/JAX vs CUDA
This is the part specs tables miss, and it's the actual deciding factor for most teams.
JAX is TPU's first-class citizen. It was built by Google specifically to trace and compile computation graphs through XLA, and it gets the newest TPU features and the best-tuned kernels first. If your stack is already JAX (increasingly common for large-scale pretraining research), a TPU is a natural target.
PyTorch on TPU goes through PyTorch/XLA, a bridge package that lowers PyTorch ops into XLA's compiled graph model. It works, and NVIDIA-scale teams do use it, but it has historically trailed native CUDA PyTorch in coverage and required more careful handling of dynamic shapes. The ecosystem's own recent direction underscores this: vLLM's 2025 TPU backend redesign explicitly routes both PyTorch and JAX models through JAX as the lowering path, because "JAX is a more mature stack" for TPU with "superior coverage and performance," according to vLLM's own writeup. Native PyTorch-on-TPU support, tracked as PyTorch/XLA RFC #9684, was still in development as of PyTorch/XLA's 2.7 release. If you live in pure PyTorch and want the least friction, that's a real point in the GPU column today.
CUDA is the default everywhere else. Every major training and inference framework — PyTorch, TensorFlow, vLLM, SGLang, DeepSpeed, TensorRT-LLM, every diffusion and video model repo on GitHub — ships a CUDA path first, often exclusively. Custom kernels (FlashAttention, fused optimizers, quantization kernels) land on CUDA first and get ported elsewhere later, if at all. This is less about which chip is faster and more about which chip your specific dependency tree already runs on without modification.
Availability: the lock-in question
This is the factor that gets underweighted. TPUs are sold exclusively through Google Cloud. There is no other cloud, no bare-metal host, and no secondary market for TPU capacity — if you commit to TPU-based infrastructure, you are committing to Google Cloud as your only vendor for that hardware, indefinitely. That's a real decision factor independent of price or performance: you lose the ability to shop providers, negotiate on price, or move a workload if Google's availability, region, or pricing changes under you.
GPUs are the opposite. H100s, H200s, and B200s are sold by NVIDIA to dozens of cloud providers, hyperscalers, and specialist GPU clouds, which means the same chip is available across many vendors at different price points and in different regions — you can compare and switch. That flexibility is the entire reason a GPU marketplace model exists at all; there's no equivalent "TPU marketplace," because there's only one seller.
Pricing: what's actually published
Google publishes TPU v6e on-demand pricing at $2.70 per chip-hour on its Cloud TPU pricing page (as of August 2026; check the live page for your region, since Google prices TPUs per-region and per-commitment-tier). That's a per-chip rate — a full pod-scale job multiplies by however many chips your slice uses.
H100 hourly pricing is set independently by each cloud and GPU marketplace rather than by NVIDIA, so there's no single number to quote — it spans roughly $2-$8+/hr depending on provider, region, and whether you're on-demand or spot, which is exactly the price spread we've written about separately. Check current cross-provider rates on the live GPU Availability Index or H100 pricing page rather than trusting a snapshot in this post — GPU prices move by the week.
The honest comparison isn't "$/hr" against "$/chip-hr" — it's cost per completed training run or cost per million tokens served, which depends on how efficiently your specific model and framework map onto each chip's compute model. Neither vendor publishes a number that settles that for you; you have to benchmark your own workload.
When a TPU actually wins
- You're training or serving a standard transformer architecture (most LLMs, ViTs, diffusion transformers) at large scale, where XLA's ahead-of-time compilation has mature kernel coverage.
- Your stack is already JAX, or you're starting fresh and willing to adopt it.
- You need the largest possible pod-scale interconnect for a single training run — Ironwood's 9,216-chip pods and 1,200 GB/s inter-chip bandwidth are built for exactly this.
- You're already committed to Google Cloud for the rest of your infrastructure and single-vendor lock-in isn't a concern.
When a GPU wins
- You need framework flexibility — custom CUDA kernels, non-standard architectures, research code that assumes dynamic shapes and control flow.
- You want multi-provider pricing and availability, or need to move a workload between clouds without touching a single-vendor hardware dependency.
- Your team is PyTorch-native and wants the deepest, most mature kernel and tooling ecosystem rather than routing through a JAX bridge.
- You're doing inference serving with unpredictable batch shapes, video/image workloads outside pure transformer math, or anything that mixes AI compute with general-purpose parallel code.
If you're evaluating hardware for a specific model and budget rather than in the abstract, the GPU recommender tool will match VRAM and throughput needs against available GPU options; for head-to-head specs on the two most commonly rented Hopper chips, see our H100 and H200 pages, or the full GPU Availability Index for current cross-provider pricing.
Frequently asked questions
Is a TPU faster than a GPU?
Neither is categorically faster — it depends on the workload and generation compared. Google's own announcement claims its current Ironwood (TPU7x) chip delivers more than 4x the per-chip training/inference performance of the prior TPU v6e generation, and its 7.37 TB/s memory bandwidth exceeds any single NVIDIA GPU's. But TPUs only run efficiently on XLA-compiled, matmul-heavy workloads with mature kernel support; a GPU running a workload outside that envelope, or one your team can tune deeper because of CUDA's ecosystem maturity, can outperform a TPU in practice even with lower peak TFLOPS.
What is a TPU in simple terms?
A TPU is a chip Google designed specifically to run the matrix multiplications behind neural networks, using a systolic array that passes data directly between compute units instead of round-tripping through memory. It's programmed through Google's XLA compiler and is available only on Google Cloud, unlike GPUs which are general-purpose processors sold by NVIDIA/AMD to many cloud providers.
Can I run PyTorch on a TPU?
Yes, through PyTorch/XLA, a bridge that compiles PyTorch code through the same XLA path JAX uses. It works for standard model architectures but has historically had less kernel coverage than PyTorch on CUDA/GPU. As of PyTorch/XLA's 2.7 release, a native (non-bridged) PyTorch backend for TPU was still in development.
Are TPUs cheaper than GPUs?
Not necessarily. Google publishes $2.70/chip-hour for TPU v6e on-demand as of August 2026; GPU pricing is set independently by dozens of providers and spans a wide range depending on model, region, and provider. The fair comparison is cost per completed job, not per-hour rate, since the two chips don't do the same amount of useful work per hour on every workload.
Do TPUs work outside Google Cloud?
No. TPUs are sold exclusively as a Google Cloud service — there's no other vendor, no bare-metal purchase option, and no secondary market. GPUs, by contrast, are available across many cloud and specialist providers, which is the basis for cross-provider GPU marketplaces.
