Stop Billing for Idle GPU Instances Automatically

Everybody who rents GPUs eventually writes the same script. Sample nvidia-smi, and if utilization has been under some threshold for some number of minutes, stop the box. It is twenty lines. It works on the first test. And within about two weeks somebody on the team has commented it out, because it killed a machine they were actively using.

That failure is not a bug in the twenty lines. It is a flaw in the premise. Zero percent GPU utilization is also exactly what a human being working looks like — editing a config, reading an error, installing a package, thinking. A GPU at idle and a GPU whose owner is mid-thought produce an identical reading. Any automation built on that one number will eventually delete somebody's afternoon, and the moment it does, it gets switched off and the idle spend comes right back.

This post is about what an automatic idle stop actually has to know to be trusted, since a stop you disable saves nothing.

TL;DR: GPU utilization alone cannot distinguish an abandoned box from an occupied one — which is why Google's own Vertex AI Workbench states plainly that "CPU usage doesn't reset the idle shutdown timer" and keys on notebook activity instead. A correct auto-stop needs a second, independent signal — is anyone actually on this machine — and a third state beyond idle and in-use: we could not read the signal. Treating an unreadable signal as "nobody's here" invents evidence and stops a box somebody is using. It also has to save before it stops, refuse to stop if the save fails, and land the box in a state you can actually resume from. Miss any of those and you have built something people turn off.

Utilization is a necessary signal and a terrible one

Start with why the naive version fails, because the failure shape explains every design decision after it.

A GPU sits at 0% during: a pip install, a dataset download, reading a stack trace, writing a prompt, a compile step, lunch, and abandonment. Only the last of those is worth stopping. The others are a person mid-task who will be furious when their environment disappears.

You can raise the timeout to compensate — act after 90 minutes instead of 20 — but that trade is worse than it looks in both directions. Long enough to be safe during interactive work is long enough that a forgotten H100 bills for another hour and a half. Short enough to catch waste is short enough to kill working sessions. There is no threshold that resolves this, because you are trying to separate two states using a measurement that does not distinguish them.

The fix is not a cleverer threshold. It is a second signal.

The second signal: is anyone actually there?

The question the utilization number is failing to answer is "is a person using this box." So measure that directly.

That means asking the machine itself: are there open interactive sessions — SSH logins, browser terminals, attached PTYs? A box with a human on a terminal is in use, whatever the GPU is doing. A box with 0% utilization and nobody connected for an hour is genuinely abandoned. Two signals separate the cases that one cannot.

This is not a niche opinion — it is what the major clouds already do, and their docs say so explicitly. Vertex AI Workbench keys its timer on notebook activity, not load: "running a cell or new output printing to a notebook is activity that resets the idle shutdown timer. CPU usage doesn't reset the idle shutdown timer." Azure ML goes further and enumerates the conditions, requiring no active Jupyter kernel sessions, no active Jupyter terminal sessions, no active runs or experiments, no VS Code connections and no custom applications running before it will call an instance inactive.

Azure's page also contains a quietly instructive admission: "we don't track VS Code SSH connections to determine activity." That is a documented blind spot in a mature product — a whole class of real user presence the detector cannot see. It is the same failure mode as a bad sshd matcher, just written down honestly. Enumerating presence is genuinely hard, and the gaps are where boxes get killed.

Two details make the difference between a presence check that works and one that quietly reports empty rooms:

Match every spelling of what you are looking for. OpenSSH 9.8 split the daemon, so a live session now shows up as sshd-session: user@pts/0 rather than the older sshd: form. A matcher keyed on the old string finds nothing on any current distribution — and reports the box as empty while somebody is typing on it. We hit exactly this.

Not every session allocates a PTY. An SSH command run without a TTY has no /dev/pts entry at all, so a probe that only enumerates PTYs misses it entirely.

Both bugs fail in the same dangerous direction: they under-report presence, which reads as "nobody is here," which stops a box in use. When you write this check, the test that matters is not "does it see me when I log in" — it is "does it see every kind of session, on the distro actually running."

Crucially, this has to be measured on the box. A control plane's own record of who has a terminal open is scoped to whichever server process happens to be handling it, and is blind to sessions established anywhere else. The agent on the machine can see all of them locally. Ask the thing that can actually see.

The third state: "we could not tell"

This is the part almost everyone gets wrong, and it is the one that decides whether the feature is safe.

A naive implementation has two states: idle and in-use. But there is always a third real-world case — the utilization reading failed, or the on-box agent is unreachable. What should the system conclude?

The tempting answer is to treat an unreachable agent as "no active session." It is the wrong answer, and it is wrong in a specific way worth naming: it invents evidence. "I could not find anyone" and "there is nobody" are different claims. Collapsing them means an agent that crashed, a network blip, or a slow sampler all silently manufacture the conclusion that the box is abandoned — and then something irreversible happens on the strength of it.

So the signal has to be three-state all the way down. Not a boolean, but present / absent / unknown, kept distinct through every layer including the database column, so nothing downstream can flatten it back. When either input is unknown, the verdict is UNKNOWN, and UNKNOWN never triggers a stop.

The general rule, which applies well beyond this feature: a signal that gates an irreversible action must be three-state, and the "we could not look" branch must block the action. A confidently wrong value is worse than a missing one, because every safety net downstream is built to respect uncertainty and to act on certainty. Give it false certainty and you have disabled the safety net.

That is how ours works — a deployment with auto-stop configured reports in-use, idle, idle past the threshold, or we can't read utilization on this box, and the last is a real state rather than a rounding of "probably idle." You can see the shape of it on the idle auto-stop feature page.

Saving is a precondition, not a step

Now the second-order problem. An automatic stop that loses your environment is not a cost saving, it is a data loss event with a discount attached. So the stop has to save first.

Which means the feature inherits every precondition of saving. If storage is not configured, the save cannot run; if the save cannot run, the stop must be refused; and if the stop is refused, the box keeps billing — the exact outcome the feature exists to prevent, except now the user has been told it is handled.

There are two rules here worth stating plainly:

Refuse rather than proceed. If the save fails, do not stop the box. Leave it running, log loudly, retry on the next pass. An unsaved stop is unrecoverable; an extra few minutes of billing is not.

Gate the opt-in where the user opts in. Do not let someone enable a protection their account cannot actually deliver. A pre-checked box promising something the system will refuse to do at the moment of truth is worse than no box at all. Check the precondition at the point of choice, not only at the point of action.

The return path is part of the feature

The last failure is the one that looks fine in testing and is infuriating in production.

A feature that stops a resource on your behalf owes you a way back — and the way back has to be reachable from the state the feature actually produces. It is entirely possible to build an auto-stop that works perfectly, saves correctly, halts the billing, and leaves the box in a status your own Resume button does not recognise. Then the user sees a stopped box, promised resumability, and no way to resume it. We shipped that and had to fix it: the discovery query that decides what is resumable had to be widened, deliberately, rather than changing what the shared close path produces — because other things land in that same terminal state and must not become resumable.

Which is the general lesson: assert the round trip, not the outgoing leg. Stop the box automatically, then start it again from the UI, and confirm your environment came back. Test the loop, not the halt.

What sensible defaults look like

For reference, ours: warn at 30 minutes idle, act at 60 minutes, with idle defined as GPU utilization under 5 percent — and every one of those adjustable per deployment after the fact from the deployment's Idle tab. The server-side default for auto-stop itself is off; in the console deploy sheet the toggle is pre-checked, and it is only offered at all when storage is configured, because of the precondition above.

The warn-then-act gap matters more than either number. A stop that happens with no prior signal feels arbitrary and gets disabled. A warning at half the interval gives whoever is actually there a chance to say so — which is precisely the human-in-the-loop backstop for the cases your two signals still cannot separate.

None of this is provider-specific. Because the mechanism is save-then-stop rather than a vendor's own suspend API, it behaves the same regardless of who is renting you the card.

Stopping the box may not stop the whole bill

One thing worth checking before you congratulate yourself on the automation: whether stopping actually stops the meter, or only the expensive half of it.

RunPod is explicit that "storage charges continue to accrue on stopped Pods" — volume disk at $0.20 per GB per month and network volume at $0.07 per GB per month, with the GPU doing nothing. AWS says the same about EBS: "some AWS resources, such as Amazon EBS volumes and Elastic IP addresses, incur charges regardless of the instance's state", and that while an instance is stopped, "charges are incurred for the storage of any Amazon EBS volumes." Google Cloud is equally direct: for a stopped instance "you continue incurring charges for any resources attached to the instance, such as disks and external IP addresses, until you delete them", and the same holds for suspended ones.

So a 150 GB volume left on a stopped RunPod pod is roughly $30 a month for a box nobody is using. That is small next to an H100 hour, and large next to zero — and it accrues silently, forever, because nothing about a stopped box prompts you to revisit it.

An auto-stop that halts compute and leaves storage running is still a large win. Just do not mistake it for reaching zero. Getting both meters to stop means the state has to live somewhere other than an attached volume, which is the same conclusion the portability argument arrives at from the other direction.

Worth knowing: most GPU marketplaces do not ship this at all

The idle-shutdown features quoted above all come from managed notebook products — Vertex AI Workbench, Azure ML, SageMaker Studio. That is not an accident; they are the surfaces where the vendor knows what "activity" means.

On the raw GPU rental side it is thinner. I could not find a documented utilization-based idle auto-stop in RunPod's pod management docs or Vast.ai's instance docs — what RunPod's docs show is a manual sleep 2h; runpodctl pod stop workaround. I checked those specific pages rather than crawling both sites exhaustively, so treat that as "not documented where I looked" rather than a guarantee of absence. But it matches the broader pattern: people hand-roll idle watchers for rented GPUs because the rental products largely do not ship one.

And where a managed product does ship it, it is usually opt-in. SageMaker Studio requires an admin to "enable idle shutdown" and enforces a minimum threshold of 60 minutes. Azure's is a toggle with a 15-minute floor. Vertex AI Workbench is the outlier that ships it on, at 180 minutes.

What a hand-rolled script gets wrong

If you are going to build this yourself, these are the five things to get right, in the order they will bite you:

1. One signal is not enough. Add presence detection, or accept that you will eventually kill a working box.
2. Handle the unreadable case explicitly. A failed sample must not read as "idle." This is one if and it is the difference between safe and dangerous.
3. Match the real system. Current OpenSSH, sessions without PTYs, container boxes with no systemd. Test against the distro you actually run.
4. Save before you stop, and refuse if the save fails. Otherwise you have automated data loss.
5. Prove you can get back. Stop it, resume it, check the environment survived. Before you trust it with anything real.

And the meta-point: the reason hand-rolled idle scripts do not stick is almost never the script. It is that stopping the box is expensive to undo. Fix the cold start and the idle policy survives contact with your team. Leave a 40-minute rebuild on the other side of the stop and everyone will find a reason to disable it — rationally. We wrote about the underlying economics in how much idle GPU time actually costs, and the full menu of stop-the-meter mechanisms in how to stop cloud GPU billing when idle. If you want to put numbers on your own situation, there is an idle cost calculator.

Common failure modes

• The watcher that got commented out. It killed a box someone was using, once. That is all it takes. The root cause is single-signal detection, not a badly chosen timeout.
• The unreachable agent that counted as absence. A monitoring failure became a stop authorisation. Three-state or bust.
• The stop that could not save. No storage configured, so the save failed and the stop was correctly refused — while the interface said the box was protected.
• The stop you cannot undo. Box halted, billing stopped, no Resume button. Technically a success, practically a support ticket.
• The threshold tuned to the wrong risk. Long timeouts protect interactive users and let forgotten boxes run. Short ones do the reverse. Neither is the fix; the second signal is.

FAQ

Can you automatically stop a GPU instance when it goes idle? Yes, and it is the single highest-value cost control for rented GPUs. The important part is how "idle" is decided: GPU utilization alone produces false positives on interactive work, so a reliable implementation also checks whether anyone is connected to the box.

Why did my idle auto-shutdown kill a machine I was using? Almost certainly because it only looked at GPU utilization. A GPU reads 0% while you edit code, install packages or read logs. Without a presence signal, those are indistinguishable from an abandoned machine.

What should the idle threshold be? With two signals, 30 minutes to warn and 60 to act is a reasonable starting point at a 5% utilization floor. With only utilization, no threshold is really safe — short values kill working boxes, long ones let forgotten boxes bill for hours.

Does stopping an instance stop the whole bill? Usually not all of it. RunPod states that storage charges continue to accrue on stopped pods, AWS charges for EBS volumes regardless of instance state, and GCP bills attached disks on stopped and suspended VMs alike. Auto-stop halts the expensive meter; reaching actually-zero means the state cannot be sitting on an attached volume.

What happens to my environment when auto-stop fires? That depends entirely on the implementation, and it is the question to ask first. A correct one saves your environment before stopping and refuses to stop if the save fails. A bare shutdown gives you back a blank box.

Does idle auto-stop work on every provider? It does if the mechanism is save-then-stop rather than a vendor-specific suspend call, since it then depends only on being able to capture the box and terminate it — which every provider supports.

Sources

• Idle shutdown (Vertex AI Workbench, Google Cloud Documentation)
• Create an Azure Machine Learning compute instance (Microsoft Learn)
• Configure idle shutdown in SageMaker Studio (AWS Documentation)
• Pods pricing (RunPod Documentation)
• Manage Pods (RunPod Documentation)
• Instances (Vast.ai Documentation)
• Amazon EC2 instance lifecycle (AWS Documentation)
• Suspend, stop, and reset instances (Google Cloud Documentation)

About the author

I am Ansh Saxena, working on Aquanode. I spend my time on the layer underneath the models — capturing GPU box state and moving it between providers. The design notes above are things we got wrong and fixed: the presence matcher that missed modern OpenSSH, the unreachable agent that was recorded as an absent user, and the auto-stopped box that had no Resume button.