Serverless GPU: What It Means and When It's Actually Cheaper

You're deciding how to serve a model: rent a GPU that stays on and bills you by the hour whether or not anyone's calling it, or use a provider that spins a container up per request and bills you by the second it's actually running. The pitch for the second option is "you only pay for what you use." That's true, but it skips the part that decides whether it's actually cheaper for you: how often "what you use" happens, and how long the GPU takes to wake up before it can use anything.

TL;DR: Serverless GPU means a provider runs your model in a container that starts on request and scales to zero when idle, billed per second of active compute instead of per hour of uptime. The catch is cold start — loading a multi-gigabyte model checkpoint into GPU memory takes real time (NVIDIA's own benchmark shows a 15GB Llama 3 8B model taking 5-48 seconds to become GPU-ready depending on storage and loading method). Serverless per-second rates run 30-80% higher than the same provider's dedicated hourly rate for the same GPU — on RunPod, $4.79/hr serverless vs. $3.29/hr dedicated for an H100 SXM, both checked August 23, 2026. Using that real premium, serverless is cheaper below roughly 69% utilization and a dedicated box is cheaper above it; the crossover point moves with whatever premium your provider actually charges.

What "serverless GPU" actually means

Serverless GPU borrows the same idea as serverless CPU compute (AWS Lambda, Cloud Functions): you don't provision a server, you deploy a function or container, the platform runs it when a request arrives, and it bills you for the seconds it ran. Applied to GPUs, that means a provider keeps a pool of GPU capacity behind the scenes and spins up a container with your model on it when a request comes in — then tears it down (or lets it sit idle briefly) when requests stop.

The three properties that define it, versus renting a GPU by the hour:

• Per-second billing on active compute, not per-hour uptime.
• Scale to zero — when nothing's running, you're not billed for the GPU sitting there (though most providers still bill a short idle-timeout tail before scale-down actually happens).
• No instance to manage — you don't SSH in, patch drivers, or babysit a box; you deploy a container image or a function and the platform handles placement.

The cold-start problem, specifically

This is the part that decides whether serverless works for your use case, and it's more specific than "it takes a second to boot." A GPU serverless cold start has to do several things in sequence before your model can answer a request: pull or already have the container image, initialize a CUDA context, and — the expensive part — get your model's weights from wherever they're stored into GPU VRAM.

NVIDIA's own benchmark of its Model Streamer tool is the cleanest primary-source number here. Loading a 15GB Llama 3 8B checkpoint (safetensors format) to GPU-ready state took:

Source: NVIDIA developer blog, Run:ai Model Streamer, checked August 23, 2026. That's for an 8B model. Bigger models — 70B and up — mean proportionally more data has to move from disk or object storage into VRAM before the first token comes out, which is why serverless providers that serve large models invest specifically in this problem rather than treating cold start as a fixed constant.

Providers have built real, measured mitigations. Modal's engineering blog reports GPU memory snapshotting cutting a vLLM-served Qwen2.5-0.5B cold start from 45s to 5s, and a ViT-plus-torch.compile cold start from 8.5s to 2.25s (Modal blog: GPU Memory Snapshots, checked August 23, 2026). RunPod's FlashBoot claims cold starts "as low as 500ms," with "95% of cold-starts less than 2.3 seconds" on their platform (RunPod FlashBoot announcement, checked August 23, 2026) — RunPod doesn't publish the mechanism behind that number on their own page, so treat it as their measured result, not a technique you can replicate elsewhere. The pattern across all of these: cold start isn't solved by "serverless" as a category, it's solved (or not) by the specific engineering a given provider has put into weight loading — ask about it directly before you commit to a platform for a large model.

What providers actually charge (checked August 23, 2026)

Real per-GPU-second and per-GPU-hour rates, pulled directly from each provider's own pricing page.

A note on Together AI: its serverless product is priced per-token for LLM inference (e.g., one current SKU runs $0.14/$0.28 per million input/output tokens), not per-GPU-second — it's a different pricing shape entirely, not directly comparable to the others in this table. Together does sell GPU-hour capacity, but as dedicated endpoints starting at $5.49/GPU-hr for H100, which is a reserved/persistent product, not scale-to-zero serverless.

Two things worth noticing in that table. First, no provider's scale-to-zero claim is quite absolute — RunPod bills a 5-second idle tail before scale-down, and Baseten explicitly counts "deploying, scaling up or down" as billed time, not just active inference. "Scales to zero" means "gets very close to zero, fast," not literally instant. Second, per-second/per-minute serverless rates for the same card run meaningfully above what the same category of provider charges for a dedicated hourly box — which is the whole basis of the break-even math below.

The break-even math, worked out

Here's the question serverless pricing pages don't answer for you: at what point does paying more per active second stop being worth it, versus just renting the box by the hour and eating the idle time?

RunPod is the cleanest source for this because it sells both products for the identical H100 SXM card, so the premium is apples-to-apples, same vendor, same hardware, checked the same day:

• RunPod Serverless, H100: $4.79/hr while active
• RunPod Pods (dedicated), H100 SXM: $3.29/hr, billed continuously whether busy or idle

Source: runpod.io/pricing, checked August 23, 2026.

Take a month of continuous availability — 730 hours, the standard way cloud providers express a 24/7 month.

Dedicated box, running the whole month regardless of load: 730 hrs × $3.29/hr = $2,401.70/month, flat, no matter how much of that time the GPU is actually doing work.

Serverless, billed only for active hours: You break even with the dedicated price when your active hours cost the same $2,401.70: $2,401.70 ÷ $4.79/hr = 501.4 active hours

501.4 hours out of 730 possible hours in the month is 68.7% utilization.

So, using RunPod's own real numbers: if your GPU is doing actual work more than roughly 69% of the hours in a month, the dedicated box is cheaper. Below that, serverless is cheaper, and the gap grows the lower your utilization goes — a workload that's genuinely busy 10% of the time pays roughly 7x less on serverless than it would leaving a dedicated H100 running 24/7 to catch occasional requests.

That 69% figure is specific to RunPod's 46% serverless premium ($4.79 vs $3.29). A provider with a smaller premium pushes the crossover higher (serverless stays cheaper for a larger share of your usage); a bigger premium pushes it lower. Compute your own crossover with (hourly dedicated rate ÷ hourly serverless rate) × 100 before you decide, using your actual quoted rates, not this example's.

Where this breaks down: bursty and unpredictable traffic

The break-even math above assumes you can accurately predict your utilization ahead of time. Two cases where it doesn't:

• Spiky traffic with a low average but hard peaks. A workload that's mostly idle but needs to handle occasional traffic spikes at full throughput can hit serverless concurrency limits or a wave of cold starts exactly when it matters most — the average-utilization math looks great, but the user-facing latency during a spike is the real cost.
• Latency-sensitive, always-on demand. If every request needs sub-second response and you can't tolerate a cold start ever, you're paying for either a dedicated box or a serverless provider's "keep warm" / minimum-instance feature — both of which erase most of the scale-to-zero savings, because you're paying to keep something running regardless.

Where a stopped-but-recoverable box fits

Aquanode isn't a serverless GPU platform — it's built around persistent, dedicated boxes: you rent a GPU, it's yours, and the software stack, weights, and configuration on it are exactly as you left them. What it does address, in a different way than scale-to-zero serverless, is the idle-cost half of this decision. When you stop a box yourself, its state is captured on the way out and can be restored — on any provider, not just the one it started on — rather than sitting there billing you at full rate while nothing runs on it. It's a different mechanism for getting near the same outcome serverless chases — near-zero cost while idle — without giving up the "it's exactly the environment I built" property that a fresh container spin-up doesn't give you.

If your traffic pattern lands above that ~69%-style crossover — genuinely busy most of the time, or bursty in a way cold starts can't absorb — a dedicated box you pause when it's actually idle is worth running the numbers on. We wrote up the mechanics of that idle-cost problem in more depth in how to stop cloud GPU billing when idle, and you can run your own utilization numbers through the idle cost calculator before deciding. Current dedicated rates by GPU are on pricing.

Frequently asked questions

What is serverless GPU?

Serverless GPU is a way of running GPU workloads — usually model inference — where a provider spins up a container on request, runs it, and scales it back down when idle, billing per second of active compute instead of per hour of standing uptime. You deploy code, not a server.

What causes GPU cold starts?

Three things in sequence: getting the container image ready, initializing a CUDA context, and loading the model's weights from storage into GPU VRAM — the last step dominates for large models. NVIDIA's own benchmark shows a 15GB model taking anywhere from about 5 to 48 seconds to become GPU-ready depending on storage type and loading method.

Is serverless GPU cheaper than a dedicated instance?

It depends entirely on utilization. Using RunPod's own published rates for the same H100 SXM card ($4.79/hr serverless vs. $3.29/hr dedicated, checked August 23, 2026), serverless is cheaper below about 69% utilization over a month, and a dedicated box is cheaper above that. Compute the same ratio with your actual provider's rates before deciding.

Which providers offer serverless GPU?

Modal, RunPod (Serverless), Replicate, Baseten, and Fal all publish per-second or per-minute GPU pricing with scale-to-zero billing as of August 2026. Together AI's serverless product is priced per-token for LLM inference rather than per-GPU-second, which makes it a different pricing shape, not a direct comparison.

Does serverless GPU really scale to zero?

Close to it, but not instantly in every case. RunPod bills a five-second idle-timeout tail before a worker scales down, and Baseten counts deploy and scale-up/down time as billed, not just active inference. Treat "scale to zero" as "near-zero, fast" rather than literally instant free idle time.

Sources

• NVIDIA developer blog: Reducing Cold Start Latency with NVIDIA Run:ai Model Streamer — Llama 3 8B, 15GB checkpoint, load-time comparison across storage types. Checked August 23, 2026.
• Modal: GPU Memory Snapshots — Qwen2.5-0.5B and ViT cold-start reduction figures. Checked August 23, 2026.
• RunPod: Introducing FlashBoot — sub-2.3s cold-start claim. Checked August 23, 2026.
• Modal pricing — per-second GPU rates. Checked August 23, 2026.
• RunPod pricing — Serverless and Pods (dedicated) rates for H100/A100. Checked August 23, 2026.
• Replicate pricing — per-second GPU rates. Checked August 23, 2026.
• Baseten pricing — per-minute GPU rates. Checked August 23, 2026.
• Fal pricing — hourly GPU rates. Checked August 23, 2026.
• Together AI pricing — per-token serverless inference and dedicated endpoint rates. Checked August 23, 2026.