Dedicated vs Shared GPU Memory: What's the Difference?

Few parts of an AI stack get misread as often as GPU memory. A team picks a GPU because the spec line says it has enough memory for their model, ships it, and then spends a week chasing latency spikes and throughput that is a fraction of what the benchmarks promised. The cause, more often than people expect, is that part of the memory their workload was using was never on the GPU at all. It was ordinary system RAM, borrowed by the driver and reached over a bus that is dramatically slower than the card's own memory.

That borrowed memory is what "shared GPU memory" means. This guide covers what each kind of memory is, how the hardware actually wires them up, why the bandwidth gap matters so much for LLM inference and training, and how to size and monitor your GPU so your model never falls into the slow tier. If you already know which model you want to run, you can jump straight to the VRAM calculator to size it.

TL;DR:

• Shared GPU memory is system RAM that the GPU borrows, usually over the PCIe bus, once its own VRAM runs out. It is not part of the card's physical memory.
• Dedicated GPU memory is the VRAM physically mounted on the card: 80GB of HBM3 on an H100, 24GB of GDDR6X on an RTX 4090.
• The bandwidth gap is the whole story. Dedicated VRAM on current data center cards runs at roughly 2 to 8 TB/s per the vendor datasheets. System RAM and the PCIe link to it run at tens of GB/s. That is a 20x to 100x difference.
• Rule of thumb: size the GPU so the model plus its working memory never spills. A card that holds your whole model beats a faster card that does not.

What is dedicated GPU memory?

Dedicated GPU memory, usually just called VRAM, is memory that lives on the GPU board itself, connected to the die over a very wide bus built to feed thousands of parallel compute units at once. Nothing else in the system competes for it.

Two families of memory technology show up as dedicated VRAM on the GPUs people rent for AI work today.

GDDR: the consumer and mid-range option

GDDR6 and GDDR6X are the memory types on consumer cards like the RTX 4090 and on mid-range data center cards like the L40S. The memory comes as individual chips placed around the GPU die on the circuit board, each wired into a shared memory bus. Per NVIDIA's own spec pages, the RTX 4090's 24GB of GDDR6X delivers about 1,008 GB/s, and the L40S's 48GB of GDDR6 delivers about 864 GB/s.

GDDR is cheap relative to its speed, which is why it dominates gaming and cost-focused inference cards. The trade-off is capacity: each package takes board space and its own traces back to the GPU, so these cards tend to top out around 24GB to 48GB. For a model that fits, GDDR is excellent value. For one that does not, capacity is the problem, not speed.

HBM: stacked memory for data center GPUs

HBM (High Bandwidth Memory) takes a different physical approach. Instead of spreading chips across the board, it stacks several memory dies vertically and connects them with through-silicon vias (TSVs), tiny vertical wires that pass straight through each die. The stack sits right next to the GPU on the same package, joined by an interposer, and the connection between them is extremely wide: thousands of bits across, compared with a few hundred for a GDDR bus. Wide and short is exactly what you want for bandwidth.

HBM2e, HBM3 and HBM3e are the generations you will meet on current data center GPUs. Per NVIDIA's datasheets and product pages:

• The A100 80GB uses HBM2e at about 2.0 TB/s.
• The H100 SXM uses 80GB of HBM3 at about 3.35 TB/s.
• The H200 uses 141GB of HBM3e at about 4.8 TB/s.
• The B200 uses HBM3e at about 8 TB/s.

HBM is the default for serious AI work because it delivers capacity and bandwidth together: 80GB to 141GB or more per GPU, moved several times faster than the fastest GDDR card.

What is shared GPU memory?

Shared GPU memory is ordinary system RAM, usually DDR4 or DDR5 attached to the CPU, that the GPU reads from and writes to when its dedicated VRAM is not enough. The GPU cannot reach that RAM directly the way it reaches its own VRAM. On a discrete card, every access has to cross the PCIe bus to the CPU's memory controller and back. On an integrated GPU, it goes through the same memory controller the CPU uses.

The driver handles this for you, which is precisely the danger: nothing fails, the job just runs much slower than it should. Shared memory shows up in a few forms.

Integrated GPUs with no VRAM at all. Intel's integrated graphics, AMD's integrated Radeon graphics and similar chips have no dedicated memory. Their only memory pool is a slice of system RAM, accessed through the CPU's memory controller. For these chips, "GPU memory" and "shared memory" are the same thing.

Discrete GPUs falling back to system RAM. On Windows, the display driver model gives every GPU a "shared GPU memory" budget, which is why Task Manager lists it next to dedicated memory. NVIDIA's Windows driver also exposes a CUDA Sysmem Fallback Policy setting that decides whether CUDA work is allowed to spill into system RAM instead of failing with an out-of-memory error. When fallback is allowed and your model is too large, it keeps running, slowly.

CUDA Unified Memory. CUDA lets a program allocate "managed" memory that both the CPU and the GPU can address with the same pointer. The driver moves pages between system RAM and VRAM on demand. When the GPU touches a page that currently lives in system RAM, it takes a page fault, and the driver migrates that page across PCIe before the GPU can continue. Used as a way to "fit" a model larger than VRAM, every one of those faults is a stall on the slow path. Many inference frameworks offer an explicit version of the same trade: offloading some layers to CPU RAM.

Virtualized and overcommitted cloud instances. In multi-tenant cloud environments, a hypervisor or virtualization layer decides how much GPU memory each tenant sees. Depending on how that layer is configured, the "GPU memory" figure an instance reports may not map one-to-one to dedicated VRAM that belongs to you alone. More on this in the cloud section below.

Every form ends in the same place: your model's bytes sit in DDR memory on the other side of a narrow link. And that link is slow. Per the PCI-SIG specification, a PCIe Gen5 x16 slot moves about 64 GB/s in each direction. A typical dual-channel DDR5 desktop or workstation setup lands around 90 GB/s, and dual-channel DDR4 around 50 GB/s (both computed below from JEDEC transfer rates). Server platforms with more memory channels per socket go higher, but they are still far from the 2 to 8 TB/s of HBM, and on a discrete GPU the PCIe link caps what the GPU can actually pull regardless of how fast the RAM behind it is.

UMA Frame Buffer Size: the BIOS setting behind integrated graphics memory

If you have ever dug around in a BIOS or UEFI menu on a machine with integrated graphics, you may have seen a setting called UMA Frame Buffer Size. UMA stands for Unified Memory Architecture. AMD boards usually use exactly that name; Intel boards tend to call the equivalent option DVMT Pre-Allocated or Graphics Memory Allocation.

It reserves a fixed block of system RAM, from a few hundred megabytes to several gigabytes, for the integrated GPU at boot. Once reserved, the operating system cannot use it. Two things about it trip people up:

• Raising it does not give you VRAM. The reserved block is still DDR4 or DDR5 on the CPU's memory controller, at the same system RAM speed. Setting it to 16GB does not turn an integrated GPU into a 16GB graphics card; it hands the GPU 16GB of memory an order of magnitude or more slower than data center HBM, and removes that 16GB from the operating system until you change it back.
• "Auto" is the right answer for almost everyone. The driver can grow the GPU's share dynamically as needed. Raising it manually only makes sense for an application that refuses to launch below a reported threshold, or an integrated-GPU gaming setup. Serious AI work is not on that list.

If you are tuning this setting because a model will not fit, the setting is not the bottleneck. The fix is a GPU with enough real VRAM, which the VRAM sizing guide walks through.

Memory bandwidth comparison

For AI workloads, bandwidth is the number to look at first: LLM inference reads the model's weights for every token, so memory speed caps token speed. Here is how the memory types compare, using only published vendor and standards figures.

Sources: NVIDIA A100 datasheet, NVIDIA H100 product page, NVIDIA H200 product page, NVIDIA L40S product page, NVIDIA RTX 4090 product page, NVIDIA DGX B200 product page (per-GPU figure is the system total divided across its 8 GPUs), Apple MacBook Pro tech specs, and the PCI-SIG PCIe 5.0 specification overview. The DDR rows are computed from JEDEC transfer rates: DDR4-3200 moves 3,200 million transfers per second across a 64-bit (8-byte) channel, so two channels give 3,200 × 8 × 2 ≈ 51 GB/s. DDR5-5600 across the same dual-channel width gives about 90 GB/s. Faster kits and server boards with more channels go higher.

Now do the division. Take DDR5 at about 90 GB/s against the H200's 4.8 TB/s and the gap is roughly 53x. Against the B200's 8 TB/s it is roughly 89x. Take the PCIe Gen5 link at 64 GB/s against the A100's 2.0 TB/s and it is about 31x. However you pair them, the slow tier is somewhere between 20x and 100x slower than the fast one. That range is not a benchmark result; it falls straight out of the spec sheets.

The practical consequence: when some of a model's bytes move from dedicated VRAM into shared system RAM, the effective bandwidth for those bytes drops from terabytes per second to tens of gigabytes per second. That is not a gentle slope. It is a cliff.

How VRAM spill affects LLM inference

The best GPU for LLM inference guide covers this in detail, but the short version is that generating tokens one at a time is memory-bandwidth-bound. For each new token, the GPU has to stream the model's active weights (plus the KV cache) out of memory before it can do the matrix math that produces the next token. At small batch sizes, the compute units spend most of their time waiting on those reads. Faster memory means more tokens per second, almost linearly.

Why it is a cliff and not a slope

Imagine a model where 90% of the weights fit in VRAM and 10% spill into system RAM. You might expect a 10% slowdown. You get something much worse, because the time per token is the sum of the time to read each part of the model, and those parts are read at very different speeds.

Say the VRAM portion reads at H100 speed, 3.35 TB/s, and the spilled portion has to cross PCIe Gen5 at 64 GB/s. The spilled bytes are read about 52 times slower. So the 10% of the model sitting in system RAM takes roughly 5 times as long to read as the 90% sitting in VRAM. The total read time per token grows to around six times what it would be if everything fit, even though only a tenth of the model moved. Push the spill to a quarter or half of the model and the slow portion dominates completely: throughput converges toward what the PCIe link alone can deliver.

That arithmetic is why the drop in tokens per second when a model spills can easily be an order of magnitude or more. The model still loads. It still answers. It just answers at a speed set by the slowest link in the chain, and that link is the PCIe bus or the DDR memory, not the GPU's own memory. The exact number you will see depends on your model, how much of it spills, your framework and your hardware, so measure it on your own setup rather than trusting a single headline multiplier.

The lesson for sizing: what matters is not how fast your fastest memory is, but how fast the memory is where your model actually lives.

Where Apple Silicon fits

Apple Silicon Macs use a different architecture. On an M-series chip, the CPU and GPU share one physical pool of memory on the same package, and both reach it at the chip's full memory bandwidth (546 GB/s on the M4 Max, per Apple's published specs). There is no PCIe hop between a "GPU side" and a "CPU side" because there is only one side.

So when a model is larger than a discrete GPU's VRAM but fits in a Mac's unified memory, the Mac avoids the PCIe transfer bottleneck entirely for the portion of the model that would otherwise need to cross it. Every byte is read at the same unified bandwidth. It is still several times slower than data center HBM, but it is a genuinely different design from PCIe-based sharing.

Impact on training workloads

Training holds far more in memory than inference: weights, gradients, optimizer state and saved activations, all resident at once.

A worked memory budget for a 70B model

This is arithmetic, not a benchmark. It comes from the model's parameter count and the documented design of the Adam optimizer, which (per the original Adam paper) keeps two running estimates, a first moment and a second moment, for every parameter it updates. Each of those is effectively another full copy of the weights in size.

For a 70B-parameter model, counting everything at 2 bytes per parameter (FP16/BF16) as a floor:

• Weights: 70B × 2 bytes ≈ 140GB
• Adam optimizer state (two moment buffers): 2 × 140GB ≈ 280GB
• Gradients (one per parameter): ≈ 140GB
• Activations: varies widely with batch size, sequence length and whether you use activation checkpointing, anywhere from tens of GB to far more

That is roughly 560GB before activations at the most optimistic precision. In practice, standard mixed-precision training keeps the optimizer moments and a master copy of the weights in FP32, which Hugging Face's memory breakdown puts at about 18 bytes per parameter in total. At that rate, 70B parameters need around 1.26TB before activations, which lines up with the roughly 9x weight footprint in our VRAM sizing guide.

Either way, no single GPU holds this. Even the H200's 141GB is a fraction of it. Full training at this scale is split across many GPUs with tensor, pipeline and data parallelism (often with sharded optimizers like ZeRO or FSDP), so that each GPU's share of weights, gradients and optimizer state fits in its own dedicated HBM.

What happens if training spills

If any piece of that budget lands in shared system RAM, every step waits on PCIe. The usual response is a smaller batch plus more gradient accumulation, which means a longer run and more billed GPU-hours for the same learning. A spill is a cost problem, not just a speed problem.

What fitting in dedicated memory buys you

When the whole training state fits in dedicated HBM, the GPU streams weights, computes gradients and updates optimizer state at full memory bandwidth, with nothing else competing for the bus. You can run larger per-GPU batches, which keeps the compute units busier. Gradient exchange between GPUs happens over NVLink, which on an H100 SXM runs at 900 GB/s per NVIDIA's spec, independent of the PCIe bus to the CPU. And step times stay predictable, which makes scaling across more GPUs behave the way you planned.

Impact on production inference

Production inference is judged on its tail. Your P50 can look great while P95 and P99 are terrible, and the tail is what users notice and SLAs are written against. Shared memory hits the tail hardest.

Why spill shows up as latency spikes

When part of a model or its KV cache lives in system RAM, some requests will need data that is not in VRAM. With Unified Memory, the GPU takes a page fault and stalls until the page arrives over PCIe. With framework-level offloading, the layers on the CPU side run at system RAM speed. Either way, those requests take a detour down the slow path.

Worse, the slow path is not even consistently slow. System RAM and the PCIe bus are shared with everything else on the host: the CPU's own work, the network stack, disk I/O, logging and monitoring agents, other processes. When the host is busy, those stalls get longer and less predictable. The result is a latency distribution with two humps: most requests come back fast, a subset takes many times longer, and some users get a long, unexplained wait.

Why dedicated VRAM keeps latency tight

When the model and its KV cache sit entirely in dedicated VRAM, every memory access the GPU makes takes the same fast, local path. There are no page faults to service, no PCIe round trips mid-request, and no contention with the host's other work. The latency distribution stays narrow, and your P99 stays close to your P50. That predictability is what a production SLA actually depends on.

Cloud GPU memory: the hidden traps

In the cloud you see a GPU name and a memory figure, not the hardware. A few patterns can make that figure differ from the dedicated VRAM your model actually gets.

Common pitfalls

Shared physical GPUs. Some cloud offerings split one physical GPU across several tenants, using hardware partitioning such as NVIDIA's Multi-Instance GPU (MIG), vGPU software, or looser virtualization that leaves tenants contending for the same resources. Even isolated partitioning like MIG gives each slice only a fraction of the card's VRAM and bandwidth. Either way, "an H100" in a listing is not necessarily a whole H100.

Blended memory numbers. A memory figure on a listing can mean different things. It can be the card's physical VRAM, or it can be a total that quietly adds some amount of system RAM the instance is allowed to borrow. A model that "fits" in the blended number on paper falls off the bandwidth cliff the moment it reaches into the borrowed portion.

Overhead you cannot use. Even on a whole, dedicated GPU, not every byte of VRAM is yours. The CUDA context, driver allocations, and any monitoring or management agents running on the host take some memory before your workload starts. It is a small share compared with the headline figure, but if you sized your model to the last gigabyte, it is enough to tip you over.

What to check before you rent

Two questions matter more than the headline memory figure:

1. Is this the card's physical, dedicated VRAM, or does the number include anything else? Match the listed figure against the vendor datasheet for that exact card and form factor.
2. Is this a whole GPU or a slice of one? If it is a partition or a shared instance, find out how memory and bandwidth are divided and what happens under contention.

Aquanode lists GPUs by their card model (H100, H200, A100, L40S and so on), so the memory you size against is that card's datasheet VRAM, and you can check it against NVIDIA's own spec page before you rent. Browse current listings on the marketplace or compare models side by side on the GPU index.

Choosing the right memory architecture by workload

Not every workload is equally sensitive to shared memory. Here is a practical way to match the workload to the hardware.

The general rule: if a workload serves real users or carries any kind of performance commitment, dedicated VRAM is not optional. Shared memory is only acceptable where speed and consistency genuinely do not matter, such as checking that a pipeline runs end to end before moving it to real hardware.

Practical workarounds and their limits

There are well-known techniques for squeezing a workload into less memory. They all help. None of them turn shared memory into dedicated VRAM.

Quantization (FP8, INT8, or 4-bit formats like GPTQ and AWQ) cuts weight memory by 2 to 4x with modest quality loss for most inference. A 70B model at 4-bit drops from about 140GB to 35-40GB and fits on a single 80GB GPU (the VRAM sizing guide has the per-model table). It does not help once the quantized model still spills: quantization changes how many bytes you have, not how fast the slow tier reads them.

Model and tensor parallelism splits a model across GPUs, combining their dedicated VRAM. It works well over a high-bandwidth interconnect like NVLink (900 GB/s per GPU on the H100 SXM), and poorly over plain PCIe or when a GPU is already spilling.

Mixed precision (BF16/FP16, or FP8 on Hopper and newer) cuts memory and usually speeds things up, but only when the result lives in fast dedicated memory.

Gradient accumulation simulates a large batch from several small micro-batches, trading a lower peak memory footprint for more, slower steps. It is a workaround for a capacity limit, not a fix.

These techniques are multipliers on good hardware. They do not rescue a workload that is running out of the wrong memory.

Monitoring GPU memory in production

Treat VRAM and memory bandwidth as first-class operating metrics, not something to check after a latency alert. The signals worth watching:

High memory bandwidth use with low compute use means the GPU is waiting on data rather than processing it: normal for small-batch inference, a warning sign when it lines up with latency spikes. NVIDIA's DCGM exposes profiling metrics for both.

Host-to-device transfer volume should be near zero in steady-state inference; sustained PCIe traffic while just serving requests is the signature of a spill. DCGM and NVIDIA Nsight Systems both surface it.

GPU page faults, if your stack uses CUDA Unified Memory, are the most direct evidence of overflow: a non-zero fault rate during steady-state serving means the model does not fully fit in dedicated VRAM. Nsight Systems traces them on both the CPU and GPU side.

VRAM used versus physical capacity, from nvidia-smi, tells you how much headroom is left before the next context or traffic spike pushes you into an out-of-memory error or, if fallback is allowed, into shared memory.

The pattern to catch early is "VRAM nearly full, memory bandwidth saturated, compute idle, PCIe busy." That combination is what shared-memory pain looks like before it reaches your users.

On Aquanode, GPU metrics gives you a live view of utilization and VRAM use on your running GPU, so you can see how close a model is running to the card's physical memory without setting up your own monitoring first.

Run on dedicated VRAM with Aquanode

The simplest way to never deal with shared GPU memory is to rent a GPU whose dedicated VRAM holds your whole model, its KV cache, and some headroom. Aquanode's marketplace lists GPUs by card model across multiple providers, so you can compare what is available and at what price in one place instead of reading each vendor's fine print.

• For production inference on mid-size and large models, the H100 (80GB HBM3) is the default choice.
• For 70B-class models, long contexts or high-concurrency serving where 80GB gets tight, the H200 (141GB HBM3e) adds capacity and bandwidth.
• For fine-tuning and training where cost per run matters, the A100 (80GB HBM2e) is still strong value.
• For development, smaller models and quantized mid-size models, the L40S (48GB GDDR6) covers a lot of ground.

Before you pick, run your model through the VRAM calculator to see how much memory it needs at your target precision and context length. Then check the live rates on the GPU index or each GPU's page. Prices move, so check current rates before you budget rather than relying on a number you saw last month.

Frequently asked questions

What happens when a model is bigger than the GPU's VRAM?

Either the allocation fails with an out-of-memory error (standard CUDA allocations on Linux, for example), or, where fallback is allowed (Unified Memory, CPU layer offloading, Windows sysmem fallback), the model loads with part of it in system RAM. The spilled portion is then read over PCIe at tens of GB/s instead of from VRAM at up to several TB/s. Because system RAM bandwidth is a small fraction of dedicated VRAM bandwidth (20x to 100x lower, per the vendor figures above), throughput can drop by an order of magnitude or more, and latency becomes much less predictable.

Is Apple Silicon's unified memory the same thing as shared GPU memory?

No. On an Apple Silicon Mac, the CPU and GPU share one physical memory pool on the same package, and both read it at the chip's full memory bandwidth (546 GB/s on the M4 Max, per Apple). There is no PCIe hop. Shared GPU memory on a PC with a discrete card is different: the GPU has to reach across the PCIe bus (about 64 GB/s on Gen5 x16) to get to system RAM. Unified memory is much faster than that kind of sharing, but still several times slower than the HBM on an H100 or H200, so it suits local work, not production serving.

How much dedicated VRAM do I need for LLM inference?

Start with parameters × bytes per parameter for the weights: an 8B model at FP16 needs about 16GB, a 70B model at FP8 about 70GB, and a 70B model at 4-bit about 35-40GB. Then add the KV cache, which grows with context length and batch size, plus roughly 10-20% for framework overhead. The goal is that the whole thing fits in dedicated VRAM with headroom to spare, with nothing spilling into system RAM. The VRAM sizing guide has the full formula and a per-model table, and the VRAM calculator does the math for you.

Can I use shared memory for AI training?

Only for trivial cases, like checking that a small model's training loop runs. Adam-based training holds several times the weight size in memory, and any spill makes every step wait on PCIe. For real training runs, use GPUs with enough dedicated HBM, such as the A100, H100 or H200, and split larger models across several of them.

How can I tell if my GPU is using shared memory?

Start with nvidia-smi and compare memory in use against the card's physical VRAM. If usage is pinned at the limit and the job is still running slowly rather than failing, suspect spill. On Windows, Task Manager's GPU tab shows dedicated and shared GPU memory separately, so non-zero shared usage for your process is a direct sign. For a closer look, NVIDIA Nsight Systems shows host-to-device transfers over time and can trace Unified Memory page faults. Sustained PCIe traffic or page faults during steady-state inference means the model is not fully in dedicated VRAM.

Why do some cloud GPUs perform worse than their specs suggest?

Usually because the GPU you are running on is not quite the GPU on the spec sheet. It might be a partition or shared slice of a physical card, with a fraction of its memory and bandwidth. The listed memory might include system RAM on top of real VRAM. Or overhead from drivers and host agents might leave you with less usable VRAM than you sized for, pushing a tightly-fit model into spill. Before renting, check that the listed memory matches the vendor's datasheet for that exact card, and ask whether you are getting a whole GPU or a share of one.