What is the CUDA Programming Model?

"CUDA" is an overloaded term. Depending on context it can mean the high-level device architecture built into every modern NVIDIA GPU, the parallel programming model designed around that architecture, or the software platform (compilers, libraries, drivers) that extends languages like C++ to expose the model to programmers. This page is about the middle one: the programming model.

NVIDIA's own vision for CUDA traces back to Lindholm et al.'s 2008 paper introducing the Tesla architecture, still the clearest primary source for a lot of the terminology and diagrams that show up in NVIDIA's current documentation. The NVIDIA CUDA C++ Programming Guide distills that vision into three abstractions a programmer actually touches:

• A hierarchy of thread groups. A CUDA kernel doesn't run as one stream of instructions, it runs as a nested hierarchy of threads: individual threads group into thread blocks, and thread blocks group into a grid spanning the whole kernel launch. Warps sit inside that hierarchy too, as the unit the hardware actually schedules.
• A hierarchy of memories. Each level of the thread hierarchy gets a matching level of memory for communicating within the group: threads in a block share fast, small on-chip memory, while the whole grid shares slower, much larger global memory. Reading from the lowest, fastest layer of that hierarchy is meant to cost close to nothing beyond executing an instruction.
• Barrier synchronization. Threads in a group can pause and wait until every other thread in the group reaches the same point, coordinating work without routing it through memory at all.

These three abstractions exist so a program written once keeps getting faster as GPUs grow, without a rewrite. A thread block is deliberately limited in how much it can coordinate with other blocks, and that limit is exactly what lets the Streaming Multiprocessor scheduler treat blocks as independent, swappable units of work. A GPU with more SMs than the one the program was originally written for can simply run more blocks side by side, and the program gets faster without a single line changing. Put provocatively: the model makes it hard to write a CUDA program that fails to benefit from a bigger GPU.

In CUDA C++, these abstractions show up as language extensions bolted onto ordinary C++. The compiler lowers that code first to PTX, a stable virtual instruction set, then to SASS, the real machine code for a specific GPU generation. A thread block, at that lowest level, becomes a cooperative thread array, the actual hardware construct an SM schedules and runs.

Getting the mapping between this model and real hardware right, rather than just getting a kernel to compile, is most of what separates a slow CUDA program from a fast one. Aquanode's marketplace rents H100, H200, B200 and MI300X instances by the hour if you'd rather test that mapping on real hardware than guess from a spec sheet.