What is a Warp?

A warp is a group of 32 threads that get scheduled together and execute in lockstep on a single Streaming Multiprocessor. When you launch a CUDA kernel, the threads it creates aren't scheduled individually: the hardware automatically groups them into warps of 32, and that grouping is the real unit of execution on an NVIDIA GPU. Every thread in a warp runs the same instruction on the same clock cycle, a model NVIDIA calls Single-Instruction, Multiple-Thread (SIMT). When threads inside one warp need to take different branches, called warp divergence, the SM has to run each branch path separately and performance drops hard. Warp size is technically an implementation-defined constant, but on every NVIDIA GPU shipped so far it's 32.

An SM almost never runs just one warp. It schedules many at once, at minimum every warp belonging to the same thread block, and that's deliberate. When a warp issues an instruction whose result isn't ready by the next cycle, most obviously a load from global memory that has to go off-chip, but also some arithmetic instructions (NVIDIA's CUDA C++ Best Practices Guide tables out the latency for each), that warp is stalled and can't issue its next dependent instruction yet. Rather than let the SM sit idle, the Warp Scheduler swaps in a different warp that already has its operands ready. This is latency hiding, and it's the mechanism GPUs lean on to turn slow memory into high sustained throughput. It's why a kernel generally runs faster when you keep enough warps resident on each SM to guarantee the scheduler always has an eligible one on hand: the fraction of cycles that actually issue an instruction is called issue efficiency, and the degree of concurrent warp scheduling behind it is called occupancy.

Warps aren't part of the CUDA programming model's official thread hierarchy of grids, blocks, and threads; they're an artifact of how NVIDIA hardware happens to implement that model. That makes them a lot like cache lines on a CPU: invisible to program correctness, but something you have to understand if you care about speed, since memory access patterns that ignore warp boundaries leave most of a warp's bandwidth on the table. The name itself comes from weaving, described in Lindholm and colleagues' 2008 Tesla architecture paper as computing's "first parallel thread technology." The same idea shows up under other names elsewhere: a subgroup in WebGPU, a wave in DirectX, a simdgroup in Metal.

How many warps an SM can hold resident, and therefore how much latency it can hide for free, varies by GPU generation; it's one of the specs worth comparing across the H100, H200 and B200 listings on the Aquanode marketplace before you fix a kernel's launch configuration.