NVIDIA GPU Architecture Explained: A Guide for AI (2026)

GPU architecture is why a chip's headline FLOPS number and its real training throughput can diverge sharply: it's the layout of the compute units, memory system, and execution pipeline that decides how much of that raw compute a given workload actually gets to use.

This page works through NVIDIA's architecture from the ground up: the building blocks inside the chip, how its memory system moves data, how NVIDIA's generations from Ampere through the upcoming Vera Rubin platform differ, and how to pick an architecture for a specific workload rather than the biggest number on a spec sheet.

What GPU Architecture Actually Is

GPU architecture describes how a chip organizes its compute units, its on-chip and off-chip memory, and the pipeline that schedules and executes instructions across all of it. It's why two GPUs with similar core counts can behave completely differently on the same job: what determines real performance is how those cores get fed, which numeric precisions they support natively, and how much data can move between memory and compute per second.

Two separate layers sit under that one term. The first is the physical microarchitecture: the actual layout of transistors, execution units, caches, and interconnect on the die. The second is the programming model, the abstraction software uses to address that hardware. NVIDIA's programming model is CUDA, which has shipped on every NVIDIA GPU since 2007 and is the reason most deep learning frameworks target NVIDIA hardware first.

Why GPU Architecture Differs from CPU Architecture

CPUs and GPUs optimize for opposite goals. A CPU is built to finish one instruction stream as fast as possible: it spends enormous transistor budget on branch prediction, out-of-order execution, and deep caches so a single thread of work completes with the lowest possible latency. A GPU gives up almost all of that per-thread sophistication and instead runs an enormous number of simple threads at once, betting that total throughput across thousands of threads beats the latency of any one of them.

The gap in raw parallelism is stark. A high-end server CPU tops out around several dozen cores; NVIDIA's H100 packs 132 Streaming Multiprocessors for a combined 16,896 CUDA Cores on a single die, per NVIDIA's own H100 architecture whitepaper.

That bandwidth gap is a large part of why GPUs run deep learning. A transformer forward pass is dominated by matrix multiplications, and matrix multiplication is exactly the dense, predictable, parallel work a GPU's memory system and Tensor Cores were built to move through as fast as possible.

The Building Blocks of NVIDIA GPU Architecture

NVIDIA organizes its chips as a strict hierarchy, and reading that hierarchy top to bottom is the fastest way to make sense of any GPU's spec sheet.

Graphics Processing Clusters

A Graphics Processing Cluster (GPC) is the largest organizational block inside an NVIDIA GPU, short of the full die itself. Each GPC bundles its own scheduling logic together with a set of Streaming Multiprocessors and a slice of cluster-level cache, and it operates largely independently of every other GPC on the chip.

GPC count is one of the main levers NVIDIA uses to turn a single physical design into a whole product line: the same die often ships with different numbers of GPCs enabled, a flagship part with every GPC active and a cheaper part on the identical die with some switched off. Part of that is a manufacturing outcome (a die with one defective GPC sells as a lower tier part instead of being scrapped), and part of it is deliberate segmentation (NVIDIA disables working GPCs on fully functional dies to create a slot below the flagship). Either way, GPC count, not the marketing name on the box, is what tells you how much of the chip you're actually getting.

Streaming Multiprocessors

The Streaming Multiprocessor (SM) is where computation actually happens. Every SM packs CUDA Cores, Tensor Cores, one or more warp schedulers, a register file, and a block of configurable shared memory, all sharing the same instruction issue logic.

SM count is the single best proxy for a GPU's raw compute budget, and NVIDIA has scaled it up steadily across generations: the A100 ships 108 SMs, the H100 SXM ships 132, and the B200 ships 160, per NVIDIA's respective architecture whitepapers. More SMs only help if a workload is large enough to keep all of them busy; a small batch size or short sequence length can leave a 160-SM chip running at a fraction of its potential simply because there isn't enough independent work to spread across it.

Inside an SM, threads execute in fixed groups of 32 called warps. A warp scheduler picks a ready warp each cycle and issues its instruction to the available CUDA or Tensor Cores. Because a memory fetch takes far longer than an arithmetic instruction, an SM keeps far more warps resident than it can execute in a single cycle, so when one warp stalls on memory, the scheduler switches to a different warp that's ready to run. That switch is the core trick behind a GPU's throughput model: it doesn't avoid memory latency, it hides it behind other work.

CUDA Cores vs Tensor Cores vs RT Cores

Every SM contains three distinct kinds of execution units, and mixing them up is the most common way to misread a GPU's spec sheet.

CUDA Cores handle general-purpose scalar arithmetic: one floating-point or integer operation per core per clock. In a training or inference run, CUDA Cores do everything that isn't a matrix multiply: activation functions, normalization layers, indexing, and data movement.

Tensor Cores are fixed-function units built for one job: matrix multiply-accumulate on small tiles of data in a single clock cycle. Where a CUDA Core computes one multiply-add, a Tensor Core computes many multiply-adds across a small matrix tile in that same cycle. Because a transformer is mostly matrix multiplications, the Tensor Core generation a chip ships with, and which numeric precisions it supports natively, is the single biggest architectural factor separating one GPU generation's AI performance from the next.

RT Cores accelerate ray-triangle intersection tests for real-time graphics rendering. They play no role in training or inference math, so they're safe to ignore entirely when evaluating a GPU for AI work. Data center parts built specifically for AI, like the A100 and H100, don't ship RT Cores at all.

GPU Memory Hierarchy

A GPU's compute units can consume data faster than any single tier of memory can supply it, so the chip layers several kinds of memory between the SM and the outside world, each one trading capacity for speed. Understanding that ladder is what turns a vague "the training run feels slow" into an actionable diagnosis.

On-Chip Memory: Registers, Shared Memory, and L1/L2 Cache

Registers sit closest to the compute units and are the fastest memory on the chip: each thread gets a private slice of the SM's register file with effectively no access latency. That file is a finite, shared resource, though. An H100 SM's register file holds 65,536 32-bit registers, split across however many threads are resident on that SM at once, so a kernel that demands too many registers per thread reduces how many threads can run concurrently.

Shared memory is a scratchpad that every thread in a block can read and write, sitting physically inside the SM with latency in the range of 20 to 30 clock cycles, far faster than a trip to main memory. A CUDA kernel typically stages data it needs to reuse into shared memory once, then has every thread in the block read it repeatedly from there instead of hitting global memory over and over. On an H100, shared memory and L1 cache share the same 228KB-per-SM physical block, and a kernel author can adjust the split between the two.

L2 cache sits one level up, shared across every SM on the die, and acts as the buffer between the SMs and the GPU's main memory. NVIDIA's H100 ships 50MB of L2; Blackwell's GB200 grows that to 126MB, roughly 2.5x more, per NVIDIA's Blackwell architecture whitepaper. A bigger L2 means more of a model's weights can stay resident in fast on-chip cache across consecutive batches instead of being re-fetched from off-chip memory on every forward pass.

Off-Chip Memory: GDDR vs HBM

Off-chip memory is where the real capacity lives, and NVIDIA uses two fundamentally different technologies depending on the product line.

GDDR, GDDR6X on Ada Lovelace parts like the RTX 4090 and GDDR7 on newer consumer cards, connects to the GPU die through a conventional memory bus. It's cheaper to manufacture and good enough for inference on small and mid-size models or for QLoRA fine-tuning, but its bandwidth becomes the limiting factor once batch sizes or model sizes grow.

HBM (High Bandwidth Memory) takes a different physical approach: it stacks multiple DRAM dies vertically and mounts that stack next to the GPU die on a shared silicon interposer, which shortens the electrical path enough to unlock dramatically more bandwidth in a much smaller footprint. That's why every data-center-class NVIDIA part from the A100 forward uses HBM rather than GDDR.

Both columns on that table matter independently for LLM work. Capacity decides whether a model and its optimizer states fit on the device at all; bandwidth decides how fast the Tensor Cores can be kept fed once they do. NVIDIA's preliminary figures for the upcoming Rubin platform point to HBM4 with a further large jump in both capacity and bandwidth, though NVIDIA has flagged the final specifications as subject to change ahead of general availability. See GPU RAM for more on how capacity and bandwidth trade off in practice, and the Aquanode GPU index for current HBM specs and pricing across the fleet.

How GPUs Execute Work

NVIDIA's execution model applies one instruction to many threads of data at once, but the way it gets there is different from how a CPU vectorizes work, and the difference matters for how you write and reason about a kernel.

SIMT vs SIMD

SIMD (Single Instruction, Multiple Data) is the classic CPU approach to data parallelism: the programmer explicitly packs several data elements into one wide register and issues a single instruction that operates on all of them at once.

NVIDIA's SIMT (Single Instruction, Multiple Threads) model arrives at a similar result from the opposite direction. A programmer writes an ordinary scalar kernel from the point of view of a single thread, with no vector packing at all, and the hardware runs that same instruction stream across an entire warp of 32 threads in lockstep. Under the hood the GPU is still executing something close to SIMD, but the vectorization is the hardware's job, not the programmer's.

That convenience has a cost. If threads inside the same warp branch onto different paths, say an if statement that some threads take and others don't, the SM can't run both paths at once: it executes each path in turn, masking off the threads that shouldn't be active in that pass. That's warp divergence, and it's one of the most common ways an otherwise reasonable-looking kernel loses performance.

The Thread Hierarchy

CUDA's thread hierarchy maps directly onto the physical hardware described above:

• Threads are the base unit: each one runs the kernel body once, against its own slice of data.
• Warps group 32 threads together as the unit the warp scheduler actually issues instructions to.
• Thread blocks group warps together; every thread in a block lands on the same SM and can share that SM's shared memory and synchronize with the rest of the block.
• Grids group every thread block launched by a single kernel call, spanning however many SMs the launch needs.

When a kernel launches, its grid is carved into thread blocks and those blocks are distributed across the GPU's available SMs. How many blocks can be simultaneously resident on one SM is capped by that SM's register file and shared memory, whichever runs out first. The ratio of warps actually resident on an SM to the maximum it could theoretically hold is called occupancy, and pushing it higher, so an SM always has a ready warp to switch to when another one stalls, is one of the main things a kernel author tunes for.

NVIDIA GPU Architecture Generations

NVIDIA ships a new GPU architecture roughly every two years, and for AI workloads the changes that matter most are Tensor Core generation, the numeric precisions supported natively, memory bandwidth, and the software features layered on top to make transformer workloads faster.

Ampere: Made Large-Scale LLM Training Practical

Ampere shipped in 2020, and its A100 flagship is what made today's scale of language model training economically realistic. It introduced third-generation Tensor Cores with native TF32 and BF16 support, structural 2:4 sparsity, which can roughly double throughput on models pruned to fit that pattern, and Multi-Instance GPU (MIG), which splits a single A100 into as many as seven fully isolated GPU instances. NVLink 3.0 connected GPUs within a node at 600GB/s of bidirectional bandwidth.

Years later, the A100 is still in wide use. Its price per hour has come down enough that it remains a sensible default for a large share of mid-scale training and fine-tuning jobs that don't need Hopper-class bandwidth. You can compare current A100 pricing on the Aquanode marketplace.

Hopper: Built for Transformers

Hopper arrived in 2022 as the first NVIDIA architecture designed around the transformer specifically, rather than general parallel compute. Its headline feature is the Transformer Engine, a hardware unit that automatically moves individual layers between FP8 and higher-precision formats at runtime, recovering most of FP8's speed without hand-tuning precision layer by layer.

The H100 paired fourth-generation Tensor Cores and native FP8 support with HBM3 memory and NVLink 4.0 at 900GB/s. The H200 followed as a memory refresh on the same compute die, swapping in HBM3e for 141GB of capacity and 4.8TB/s of bandwidth, which made it the natural choice for inference workloads with long context windows.

The H100 remains the default GPU for production training in 2026: broad framework support across PyTorch, JAX, and every major inference server, combined with well-understood performance characteristics, makes it the safe first choice for most teams. You can compare current H100 and H200 pricing directly on the Aquanode marketplace.

Ada Lovelace: The Consumer and Inference Line

Ada Lovelace runs alongside Hopper as NVIDIA's consumer and workstation architecture, and it trades HBM for GDDR6X, which caps both its bandwidth and its capacity well below the data-center line.

The L40S is the data-center-oriented card in the family and has become a common choice for inference serving, where its lower cost per hour outweighs its lower bandwidth. The RTX 4090 is popular for local fine-tuning of smaller models. Neither is a realistic choice for training a large model from scratch, since HBM bandwidth, or the lack of it, is the binding constraint at that scale.

Blackwell: The Current Flagship

Blackwell launched in 2024 and is NVIDIA's current top-of-line data center architecture. It brought fifth-generation Tensor Cores with native FP4 and FP6 support, a dual-die design that fuses two reticle-limited chiplets into a single logical GPU carrying 208 billion transistors connected by a 10TB/s die-to-die link, 180GB of HBM3e at 8TB/s, and NVLink 5 at 1.8TB/s per GPU.

The GB200 superchip pairs two B200 GPUs with a Grace CPU over NVLink-C2C at 900GB/s, removing PCIe entirely from the CPU-GPU data path. At FP4 precision, NVIDIA has published inference cost reductions of up to 25x relative to Hopper running FP16 on comparable models. Stacked at rack scale, the GB200 NVL72 links 72 B200 GPUs and 36 Grace CPUs into what behaves, from a software perspective, like one enormous accelerator rather than 72 separate machines. You can rent B200 instances alongside Hopper and Ampere hardware on the Aquanode marketplace, or compare all of them at once with the GPU recommender.

What's Next: Vera Rubin

Vera Rubin is NVIDIA's next architecture, previewed at CES 2026 as the successor to Blackwell. Per NVIDIA's own announcement, the platform moves to a full node shrink on a dual-die 3nm design, adds a next generation of Tensor Cores, and pairs with 288GB of HBM4 memory and NVLink 6 for GPU-to-GPU communication. The Vera Rubin Superchip combines two Rubin GPUs with a custom Vera Arm CPU in one package, and NVL72 rack configurations are reportedly already reaching early hyperscaler partners, with broader enterprise cloud availability expected afterward.

NVIDIA has been explicit that exact performance and memory bandwidth figures for Rubin remain preliminary ahead of general availability, so treat any specific number attached to it today as directional rather than final.

GPU Architecture Generation Comparison

NVIDIA vs AMD GPU Architecture

NVIDIA holds the large majority of AI training infrastructure in 2026, but AMD has carved out real share where raw cost per FLOP and memory capacity matter more than ecosystem maturity.

CDNA vs CUDA

NVIDIA runs one unified architecture line across graphics and compute: every NVIDIA GPU, consumer or data center, executes the same underlying CUDA model. AMD instead splits its line in two. RDNA is the graphics-focused consumer architecture, and CDNA (Compute DNA) is a separate, compute-only architecture built for AI and HPC, with no display output or ray-tracing hardware at all. AMD's MI300X is a CDNA 3 part built on that dedicated line.

Where the MI300X pulls ahead on paper is memory: it ships 192GB of HBM3 at 5.3TB/s, more capacity than an H100 and more bandwidth than an H200, which matters for serving models that barely fit in memory on other cards. AMD connects the MI300X's multiple compute and memory dies internally over its own Infinity Fabric interconnect.

The gap that keeps most teams on NVIDIA hardware isn't the silicon, it's software. CUDA has a two-decade head start and is the default backend nearly every training and inference framework was built and tuned against first. AMD's ROCm stack has closed much of that gap, but it typically still costs more engineering time to port to than simply picking a different NVIDIA instance. Aquanode lists MI300X instances alongside NVIDIA hardware for teams that have already made that port.

Multi-GPU Scaling

A single GPU, no matter how large, eventually runs out of room: either a model doesn't fit in one GPU's memory, or a single GPU's throughput is too slow to hit your training timeline. Past that point, how efficiently GPUs talk to each other decides how much of a second, third, or seventy-second GPU's compute you actually get to use.

NVLink and NVLink Switch

NVLink is NVIDIA's proprietary GPU-to-GPU interconnect, built to move data directly between GPUs without routing it through the PCIe bus and the CPU. Each generation has roughly doubled bandwidth: NVLink 4.0 on Hopper delivers 900GB/s of bidirectional bandwidth per GPU, and NVLink 5.0 on Blackwell doubles that again to 1.8TB/s.

NVLink Switch is what extends that fabric beyond a single node. Early NVSwitch generations connected the 8 GPUs on one server baseboard into a crossbar; Blackwell's rack-scale NVLink Switch goes further, linking all 72 GPUs in a GB200 or Rubin NVL72 rack into one non-blocking fabric where any GPU can reach any other GPU at full bandwidth. That's the mechanism that lets a 72-GPU rack behave like one very large accelerator instead of 72 servers that happen to share a rack. See NVLink vs PCIe and NVLink vs InfiniBand for how NVLink compares to the interconnects it replaces at each scale.

Choosing a GPU Architecture for a Workload

Picking an architecture is a matching problem, not a search for the single most powerful part on the page. Four dimensions decide whether a given GPU actually fits a given job: VRAM capacity (does the model and its optimizer state fit at all), memory bandwidth (how fast the Tensor Cores stay fed once it does), native precision support (whether the Tensor Core generation covers the formats your training or inference stack actually uses), and interconnect generation (how much multi-GPU scaling costs you once one GPU isn't enough).

None of these mappings are absolute; a small team can train a 70B model on Ampere hardware given enough GPUs and time, it just costs more wall-clock time and more inter-node bandwidth to do it. The Aquanode GPU recommender matches a workload description to current pricing across every architecture in this table, and the full GPU index lists what's available right now, including on-demand pod pricing per generation.

Closing Thoughts

NVIDIA GPU architecture is a stack of decisions layered on top of each other: the GPC-to-SM-to-core hierarchy decides how much work runs at once, the memory hierarchy from registers up through HBM decides how fast data reaches that compute, and each new generation adds precision formats and interconnect bandwidth that compound into a real difference in AI throughput, not just a bigger number on a spec sheet.

Once you know which of those layers is actually limiting your workload, the right GPU choice mostly picks itself.

FAQ

What is GPU architecture?

GPU architecture is the design of a GPU's compute units, memory system, and execution pipeline, and how those pieces are wired together. It determines what kinds of math the chip accelerates, how much memory it has and how fast that memory moves, and which numeric precisions its hardware supports natively.

What's the difference between a Streaming Multiprocessor and a CUDA Core?

A Streaming Multiprocessor is the self-contained execution unit that owns a warp scheduler, a register file, shared memory, and a pool of CUDA and Tensor Cores underneath it. A CUDA Core is one of the individual scalar arithmetic units inside that pool. NVIDIA's H100, for instance, packs 128 CUDA Cores per SM across 132 SMs, for 16,896 total on the die.

What's the difference between Ampere, Hopper, and Blackwell?

Ampere introduced TF32 and BF16 Tensor Cores and made today's scale of LLM training economically viable. Hopper added the Transformer Engine and native FP8 support alongside higher HBM3 bandwidth, purpose-built for transformer workloads. Blackwell added FP4 and FP6 precision, a dual-die 208-billion-transistor design, and 8TB/s of HBM3e bandwidth on the B200. NVIDIA has published inference throughput gains of up to several times over Hopper for large mixture-of-experts models run at FP4.

What is Vera Rubin and when is it available?

Vera Rubin is NVIDIA's next GPU platform after Blackwell, previewed at CES 2026. It's expected to ship 288GB of HBM4 memory, NVLink 6, and a new generation of Tensor Cores, packaged with a custom Vera Arm CPU in the Vera Rubin Superchip. NVIDIA has said final specifications remain preliminary, and broader enterprise cloud availability is expected after the initial hyperscaler rollout.

Does GPU architecture matter more than VRAM for AI workloads?

They're independent constraints, not competing ones. VRAM capacity is a hard gate: if a model and its optimizer states don't fit, training simply fails, regardless of how fast the architecture underneath it is. Architecture, meaning Tensor Core generation and memory bandwidth, then decides how fast training or inference runs once the model does fit. An H100 only beats an A100 on a given job if that job's software stack actually uses FP8; if it doesn't, the newer architecture buys you less than the spec sheet implies.