What is a Streaming Multiprocessor?

A Streaming Multiprocessor (SM) is the closest thing an NVIDIA GPU has to a CPU core: it executes instructions and holds state for that execution in a bank of registers and caches. But the resemblance stops early. An SM is a much simpler, weaker processor than a modern CPU core. Instructions are pipelined within the SM the way they have been on CPUs since the 1990s, but there's no speculative execution and no branch-prediction hardware guessing which way an if will go. What an SM gives up in per-thread sophistication, it makes up in sheer count of threads running side by side.

The scale difference is stark next to a high-end CPU. An AMD EPYC 9965, drawing up to 500W, packs 192 cores that can each run two threads at a time, for 384 threads in flight at roughly 1.25W per thread (per AMD's own spec sheet). An H100 SXM, drawing up to 700W, has 132 SMs, and each SM has four Warp Schedulers that can each dispatch an instruction to 32 threads (a warp) every clock cycle. That works out to over 16,000 threads making genuine forward progress every single cycle, at about five centiwatts apiece, according to NVIDIA's H100 datasheet.

SMs also oversubscribe well past that number. A single H100 SM can hold up to 2048 threads concurrently, split across 64 groups of 32, so the full chip juggles more than a quarter million concurrent threads even though only a fraction execute on any given cycle. The rest are simply waiting their turn, and switching which group runs next costs almost nothing: because every thread's registers live in a dedicated slice of the SM's register file rather than being saved to and restored from memory, a warp swap takes a single clock cycle, over a thousand times faster than a full context switch on a CPU. GPU L1 caches are also programmer-managed and shared across the warps scheduled together onto one SM, so unlike a CPU context switch, swapping warps doesn't tank the cache hit rate either.

That cheap, instant swapping is the whole trick behind GPU throughput. Instead of stalling on a slow memory fetch, the Warp Scheduler just issues to a different warp that's already got its operands ready, keeping the SM's pool of CUDA Cores and Tensor Cores fed. CPUs chase the same goal, hiding latency, with large hardware-managed caches and branch prediction, which eats into the silicon and power budget left over for actual computation. For workloads like matrix multiplication or transformer inference, where a programmer can predict and stage the data a kernel will reuse (see Shared Memory), the GPU's approach to hiding latency wins by a wide margin.

Each SM sits inside a larger Graphics Processing Cluster, and it's the count and generation of SMs on a die, more than clock speed alone, that mostly explains why one card in the Aquanode marketplace is priced above another.