GPU Glossary
Plain-language definitions for the CUDA and GPU architecture terms in specs, benchmarks and vLLM docs: cores, warps, memory hierarchy and interconnects.
Device Hardware
Core
A GPU core is one of the compute units packed inside a Streaming Multiprocessor. What separates a CUDA Core from a Tensor Core, and why the CPU-core analogy is misleading.
CUDA Core
A CUDA Core is the unit inside a Streaming Multiprocessor that executes scalar arithmetic, one instruction issued to a whole group at a time. What separates it from a Tensor Core, whether more of them means a faster GPU, and where they fit in AI training and inference.
GPU RAM
GPU RAM is the large off-die memory pool every Streaming Multiprocessor shares, built from slower, denser DRAM cells rather than the SRAM used in registers and cache.
Graphics/GPU Processing Cluster (GPC)
A Graphics/GPU Processing Cluster (GPC) groups Texture Processing Clusters and a raster engine into one of the largest physical units on an NVIDIA die. What changed with H100-class thread block clusters.
HBM (High Bandwidth Memory) (HBM)
HBM is stacked DRAM packaged beside a GPU die, giving data center cards several TB/s of memory bandwidth. Why LLM inference depends on it, and HBM vs GDDR.
Register File
The register file is the fastest, closest-to-the-core memory on a Streaming Multiprocessor. How it's organized, what backs it in PTX, and why using too much of it per thread quietly kills occupancy.
Streaming Multiprocessor (SM)
A Streaming Multiprocessor is the closest thing an NVIDIA GPU has to a CPU core, but far simpler and far more numerous. How SMs trade per-thread sophistication for massive parallelism and near-free context switching.
Tensor Core
A Tensor Core is the GPU hardware unit that executes an entire matrix multiply-accumulate as one instruction instead of one scalar multiply at a time. How that trade unlocks NVIDIA's highest FLOP counts, and why an H100 has only four of them per SM.
Unified Memory
Unified memory lets the CPU and GPU share one memory pool, as in Apple M-series chips and NVIDIA Grace Hopper. It adds capacity, not bandwidth.
VRAM
VRAM is the memory attached to a GPU that holds the data it works on, and it caps which AI models fit. VRAM vs RAM, how to check yours, and how much AI needs.
Warp Scheduler
The Warp Scheduler decides which warp of threads runs next on a Streaming Multiprocessor, every single clock cycle. Why that decision is nearly free on a GPU and expensive on a CPU.
Device Software
CUDA Programming Model
The CUDA programming model organizes GPU code around a nested hierarchy of threads and memory. The three abstractions from NVIDIA's own programming guide, and why they let one program get faster on every new GPU without a rewrite.
Kernel
A CUDA kernel is the function a GPU programmer writes and launches, executed once per thread across thousands of threads at once. How kernels map onto the thread and memory hierarchy, with two worked matrix-multiply examples.
Shared Memory
Shared memory is the fast, on-chip pool of memory a CUDA thread block uses to avoid repeatedly hitting slower global memory. The standard load-compute-store pattern it enables, and where bank conflicts come from.
Thread Block
A CUDA thread block is the smallest unit of thread coordination a programmer directly controls, sitting between a kernel grid and a single thread. How blocks are sized and why they must run independently of each other.
Warp
A warp is a group of 32 threads that a GPU schedules and executes together in lockstep. What warp divergence costs, how warps enable latency hiding, and why they sit outside the official CUDA thread hierarchy.
Host Software
cuBLAS
cuBLAS is NVIDIA's tuned, closed-source implementation of the BLAS standard for GPUs, and the default engine under most matrix math in PyTorch and similar frameworks. The column-major memory layout quirk that trips up almost everyone who calls it directly.
CUDA
CUDA (Compute Unified Device Architecture) is NVIDIA's platform for running general-purpose code on its GPUs, and the layer most AI frameworks build on.
cuDNN (CUDA Deep Neural Network library) (cuDNN)
cuDNN is NVIDIA's GPU library of tuned deep learning operations such as convolution, attention and normalization. PyTorch and JAX call it for you.
CUTLASS
CUTLASS is NVIDIA's open-source C++ template library for building custom high-performance GEMM kernels, the toolkit underneath many of the fastest published matrix-multiply implementations. How it differs from calling a fixed routine in cuBLAS.
MLX (MLX)
MLX is Apple's array framework for machine learning on Apple silicon. MLX-format models run on Macs, not on rented NVIDIA GPUs.
NCCL (NVIDIA Collective Communications Library) (NCCL)
NCCL is NVIDIA's library for all-reduce and other multi-GPU communication. PyTorch uses it to sync GPUs, and NVLink vs PCIe decides how fast it runs.
Nsight Systems
NVIDIA Nsight Systems is a profiling and tracing tool for CUDA programs that lays CPU and GPU activity out on one timeline. Why almost every serious CUDA project ends up needing it.
nvidia-smi
nvidia-smi is the command line tool for querying and managing NVIDIA GPUs, built on the NVML management library. What it reports, what it can change, and why its text output isn't a stable interface.
Performance
Arithmetic Intensity
Arithmetic intensity is the ratio of compute operations to bytes moved in a kernel. Why it decides whether a workload is compute-bound or memory-bound, and how tricks like recomputation trade memory traffic for extra FLOPs.
Bank Conflict
A bank conflict is what happens when threads in a warp hit the same shared memory bank from different addresses, forcing the hardware to serialize accesses it could otherwise run in parallel.
Compute-bound
A compute-bound kernel is limited by arithmetic throughput rather than memory bandwidth. When LLM inference hits this regime, and a back-of-the-envelope estimate for the batch size it takes to get there.
Memory Coalescing
Memory coalescing is a hardware technique that folds several threads' logical memory reads into one physical DRAM access. Why it exists, how it maps onto a warp, and a benchmark showing what a bad access pattern costs.
Roofline Model
The roofline model plots a kernel's arithmetic intensity against two hardware ceilings, memory bandwidth and arithmetic bandwidth, to show at a glance whether it's compute-bound or memory-bound. Where it came from and why GPUs need it.
TFLOPS
TFLOPS means trillions of floating-point operations per second, a GPU's peak math speed. Why precision changes the number and why it won't predict LLM speed.
Architecture
GPU Architecture
How NVIDIA GPUs are actually built, from Graphics Processing Clusters and Streaming Multiprocessors down to the memory hierarchy, and how Ampere, Hopper, and Blackwell differ.
RoPE scaling
RoPE scaling rescales rotary position embeddings (linear, YaRN) so an LLM can handle prompts longer than its training context, at a KV-cache cost.
Interconnects
NVLink vs InfiniBand
NVLink connects GPUs inside one server; InfiniBand connects servers to each other. Where the two interconnects overlap, where they don't, and why big clusters run both.
NVLink vs PCIe
NVLink and PCIe both move data in and out of a GPU, but at very different scales. Where each interconnect wins on bandwidth, latency, cost, and compatibility.
NVSwitch
NVSwitch is NVIDIA's switch chip for NVLink. It connects every GPU in a server to every other at full NVLink speed, which multi-GPU training and serving need.
RDMA (Remote Direct Memory Access) (RDMA)
RDMA lets one machine read or write another's memory directly through the network adapter, skipping the remote CPU. It is what multi-node GPU training runs on.
Scheduling
AI Workloads
Abliteration
Abliteration edits a language model's weights to remove its refusal direction, with no training run. How it works and what it means for GPU memory.
Data Parallelism
Data parallelism copies the whole model onto every GPU, splits each batch between them and averages gradients with an all-reduce. Simple, but memory-hungry.
Distillation
Distillation trains a small student model to imitate a large teacher, so you get much of the quality at a fraction of the GPU memory.
FlashAttention
FlashAttention is an exact attention algorithm that tiles the computation so the full attention matrix is never written to GPU memory, cutting HBM traffic.
FSDP (Fully Sharded Data Parallel) (FSDP)
FSDP shards a model's weights, gradients and optimizer state across GPUs and gathers each layer only when needed, so models too big for one GPU can train.
GGUF (GGUF)
GGUF is a single-file binary format that packs a model's weights and metadata together, used by GGML-based runtimes such as llama.cpp.
KV Cache
A KV cache stores the key and value tensors of past tokens so an LLM never recomputes them. It grows with context length and batch size, and it eats VRAM.
LoRA (LoRA)
LoRA fine-tunes an LLM by freezing its weights and training small low-rank matrices added to its layers, cutting trainable parameters and GPU memory.
Mixture of Experts (MoE) (MoE)
A Mixture of Experts model routes each token to a few expert sub-networks, so compute follows active parameters but VRAM must hold every expert.
Model merging
Model merging combines the weights of several fine-tuned models into one, with no extra training and no extra inference cost. Methods and GPU needs.
Pipeline Parallelism
Pipeline parallelism puts different layers of a model on different GPUs and streams micro-batches through them. It needs little bandwidth but leaves idle gaps.
QLoRA (QLoRA)
QLoRA fine-tunes an LLM by freezing its weights in 4-bit and training small LoRA adapters on top, so a 65B model tunes on one 48GB GPU.
Quantization
Quantization stores a model's weights, and sometimes activations, in lower-precision formats like INT8 or 4-bit, cutting VRAM use for a small accuracy cost.
Speculative Decoding
Speculative decoding has a small draft model guess several tokens that the large model checks in one pass, so LLM output is faster with identical results.
Tensor Parallelism
Tensor parallelism splits each layer's matrix multiplications across several GPUs that work on every token together. It needs NVLink-class bandwidth.
Numeric Formats
AWQ (AWQ)
AWQ (Activation-aware Weight Quantization) compresses LLM weights to low-bit integers by protecting the channels that activations show matter most.
BF16 (bfloat16) (BF16)
BF16 is a 16-bit float with FP32's 8 exponent bits but only 7 mantissa bits. It is the default for training and costs 2 bytes per model parameter.
EXL2 (EXL2)
EXL2 is the quantization format of ExLlamaV2: GPTQ-based weights with mixed bit widths so a model can hit any average from 2 to 8 bits.
FP4 (NVFP4 and MXFP4) (FP4)
FP4 is a 4-bit floating-point format that stores model weights at 0.5 bytes each. NVFP4 and MXFP4 add block scaling, and native FP4 math needs Blackwell GPUs.
FP8 (FP8)
FP8 is an 8-bit floating-point format (E4M3 and E5M2) that halves memory versus 16-bit while keeping an exponent, used to run LLMs on newer GPUs.
GPTQ (GPTQ)
GPTQ is a one-shot post-training method that compresses LLM weights to 3 or 4 bits using approximate second-order information.
Building on GPUs? Aquanode runs the workload.
Deploy on H100, H200, B200, A100 and MI300X across a multi-provider marketplace, without racking your own hardware or committing to one cloud's spec sheet.
