What is a CUDA Core?

A CUDA Core is the part of a Streaming Multiprocessor that actually executes scalar arithmetic: an add, a multiply, a fused multiply-add, one instruction at a time. Contrast that with a Tensor Core, which is wired to run an entire small matrix multiply-accumulate in a single instruction. Both sit inside the same SM, and a GPU's real throughput on a given workload depends on how that workload splits between the two.

Unlike a CPU core, a CUDA Core doesn't fetch or schedule its own instructions. That's the job of the Warp Scheduler, which issues one instruction to a whole group of CUDA Cores at once, and every core in the group applies that instruction to its own registers. The group size is normally 32, the width of a warp, though some recent architectures can issue to smaller groups at a cost to efficiency. This is exactly why a CUDA Core count on a spec sheet describes lockstep capacity, not the kind of independent processing power a CPU core count implies.

The term is also less fixed than it sounds. What actually sits inside a "CUDA Core" changes across GPU architecture generations: a different mix of 32-bit integer units and 32-bit or 64-bit floating point units gets grouped under the same label from one generation to the next. NVIDIA's H100 whitepaper, for instance, lists 128 "FP32 CUDA Cores" per SM, a count of the 32-bit floating point units specifically, which happens to be double the number of 32-bit integer or 64-bit floating point units on the same chip. If you're estimating real throughput for a specific precision, count the hardware units for that operation directly rather than trusting one headline "CUDA Core" number.

What CUDA actually stands for

CUDA stands for Compute Unified Device Architecture, NVIDIA's parallel computing platform and instruction set. It's easy to conflate "CUDA" the platform with "CUDA Cores" the hardware, but they're different layers. CUDA Cores are physical silicon that runs regardless of what software touches it, while the CUDA platform, meaning the compiler, driver, runtime, and libraries, is what a programmer uses to write the kernels that get scheduled onto those cores. Before CUDA existed, in the mid-2000s, that same physical hardware was mostly reachable only through graphics APIs, so a scientist who wanted to run linear algebra on a GPU had to disguise the problem as a rendering pass. CUDA is what turned a texture-mapping pipeline into a general-purpose parallel computer, and it's the reason a CUDA Core, despite the name, has nothing to do with graphics rendering specifically: it just executes whatever scalar instruction stream the platform hands it, rendering included.

That platform-versus-hardware split is also why CUDA Cores are NVIDIA-specific. AMD and Intel GPUs have their own scalar execution units and their own compute stacks (ROCm and oneAPI, respectively), but code written for CUDA doesn't run on them without a translation layer, since "CUDA Core" describes a piece of silicon that only exists inside a chip NVIDIA built to run CUDA's instruction set.

How many CUDA Cores does a GPU actually have?

Core counts scale enormously across a single product line, which is part of why the number gets quoted so often as a marketing headline. A few widely published spec-sheet figures:

Two things stand out from that table. First, a data center GPU like the A100 can carry noticeably fewer CUDA Cores than a consumer flagship, because the data center part trades some scalar core count for a much larger Tensor Core allocation and far more memory bandwidth, the two things that actually decide AI training throughput. Second, the jump from A100 to H100 SXM is driven mostly by more SMs on a larger die, not a redesigned core, which is consistent with CUDA Core counts being an SM count multiplied by a fixed per-SM core count for a given architecture.

CUDA Cores vs. Tensor Cores

Every NVIDIA GPU since the mid-2000s has shipped CUDA Cores. Tensor Cores didn't arrive until the Volta generation in 2017, purpose-built for the matrix-heavy math behind deep learning. The two aren't competing designs so much as a generalist and a specialist sharing a die:

In a training or inference run, the split shows up directly. Tensor Cores carry the large matrix multiplications inside attention and feed-forward layers, while CUDA Cores handle everything around them, elementwise activation functions, normalization, indexing, and data layout, that doesn't fit the fixed matrix-multiply shape a Tensor Core expects.

Does a higher CUDA Core count mean a faster GPU?

Not reliably. CUDA Core count only predicts performance for the slice of a workload that is compute-bound on scalar FP32 or INT32 math and can actually keep every core fed, which is narrower than the spec sheet number suggests. An RTX 3090 lists more CUDA Cores than an RTX 4080, yet the 4080 wins many real benchmarks, because architecture, clock speed, cache size, and memory bandwidth all move the ceiling too. The pattern is sharper in AI workloads: an A100 has fewer CUDA Cores than several consumer cards, but its Tensor Core throughput and HBM capacity make it the faster choice for training, because that's where the workload's arithmetic intensity actually lands.

What else decides real throughput, alongside raw core count:

• Memory bandwidth between GPU RAM and the SMs
• Cache and shared memory capacity, which decide how often a core stalls waiting on a load
• Clock speed and per-core FMA throughput
• Tensor Core generation and supported precisions, for anything matrix-heavy
• Whether the warp scheduler can keep the core group's occupancy high given a kernel's register usage

CUDA Cores in AI and machine learning

CUDA Cores aren't idle during AI training and inference, even on GPUs whose marketing leads with Tensor Core numbers. A training step still needs data preprocessing, gradient bookkeeping, optimizer updates, and non-matmul activation functions, and all of that scalar work runs on CUDA Cores. Kernels written against CUTLASS or cuBLAS route the heavy matrix math to Tensor Cores automatically and lean on CUDA Cores for the surrounding glue, which is why a healthy training run keeps both busy rather than one alone.

Walking through a single training step makes the split concrete. Loading and normalizing a batch, and any data augmentation applied to it, runs as scalar work on CUDA Cores before a single matmul happens. The forward pass then alternates: Tensor Cores handle the big matrix multiplies inside attention and feed-forward layers, and CUDA Cores handle everything between them, softmax, layer normalization, residual adds, and activation functions like GELU that don't reduce to a matrix multiply. Backpropagation repeats that same split computing gradients, and the optimizer step at the end (updating every weight from its gradient, momentum term, and learning rate) is scalar, elementwise work that again lands entirely on CUDA Cores.

For inference, particularly at small batch sizes, a request can be too small to fill the Tensor Cores, so CUDA Cores end up carrying more of the proportional work than a spec sheet would suggest. That's one reason a raw Tensor Core FLOPs number overstates real-world serving throughput at low concurrency, and why a model that looks compute-bound on paper can still be memory- or scheduling-bound in a live serving queue.

Picking hardware by CUDA Core count alone will mislead you either way. Aquanode's marketplace lists CUDA Core and Tensor Core counts side by side for each listed H100, H200, and B200 instance, so you can weigh both against the shape of your own workload before renting.

FAQ

What does CUDA stand for? Compute Unified Device Architecture, NVIDIA's parallel computing platform. The name predates, and is distinct from, "CUDA Core," the hardware unit that runs CUDA-compiled instructions.

Are CUDA Cores the same as Tensor Cores? No. CUDA Cores execute general scalar arithmetic one operation at a time across a group; Tensor Cores execute a small matrix multiply-accumulate in a single instruction. See the comparison table above.

Does a GPU with more CUDA Cores always perform better? No. Core count only matters for the portion of a workload it actually keeps busy. Memory bandwidth, cache, clock speed, and Tensor Core capability usually decide real-world throughput more than the CUDA Core count on a spec sheet.

Which GPUs have CUDA Cores? Every NVIDIA GPU built since the original CUDA-capable hardware in the mid-2000s, from consumer GeForce and workstation RTX cards to data center GPUs like the A100, H100, and B200. AMD GPUs use a separate compute stack and don't run CUDA at all.

What languages can target CUDA Cores? CUDA C++ is the primary language, with support for C and Fortran as well. Python developers usually reach CUDA Cores indirectly, through PyTorch, JAX, or a wrapper library like PyCUDA, rather than writing raw kernels by hand, and libraries like cuBLAS and CUTLASS cover most of the matrix-heavy work without you writing a kernel at all.

How many CUDA Cores do I need for training or inference? There's no fixed number, since core count alone doesn't determine throughput for AI workloads. A GPU with a few thousand CUDA Cores handles small fine-tuning jobs and light inference fine; large-scale pretraining is usually bottlenecked by Tensor Core throughput, memory bandwidth, and total memory capacity well before CUDA Core count becomes the limiting factor.

Do CUDA Cores matter when renting a cloud GPU? They're one input, not the whole story. Weigh CUDA Core count alongside Tensor Core generation and memory bandwidth, especially for anything matrix-multiply heavy, using the GPU recommender to match a listed instance to your workload rather than picking by core count alone.