Snapshot a GPU Environment, Migrate to Another Cloud

The plan sounds simple. Snapshot the GPU box you have been working on, take the snapshot to a cheaper or more available provider, restore it, carry on. Every cloud has a snapshot button, so this should be a solved problem.

It is not solved, and the reason is worth understanding before you try it: every provider-native snapshot mechanism is deliberately scoped to that provider, usually to a single region and sometimes to a single physical machine. They are not built to let you leave. Then, even if you get the bytes across, there is a second trap in the CUDA driver stack that turns a perfect copy into a box that will not run.

This post is the mechanics of doing it anyway — what has to be captured, what breaks on the far side, and how to sequence it.

TL;DR: An AWS EBS snapshot must be explicitly copied to reach another region. A GCP snapshot's storage location cannot be changed after creation. A RunPod network volume pins your workers to one datacenter and does not sync between volumes. A Vast.ai volume "cannot be moved or attached to instances on other machines" at all. So a portable migration cannot use the provider's snapshot — it needs a capture of the filesystem itself, plus a restore that rebuilds anything welded to the old host's driver version.

Why the provider's snapshot will not travel

Start with the constraint, because it decides the whole design. Each of these is the vendor's own documentation.

AWS EBS snapshots are region-scoped. Getting one to another region is an explicit copy operation that re-transfers the data: "You can use the copy-snapshot command to copy a snapshot from the source Region to the current Region... The first time you copy an EBS snapshot to a particular Region, all of the data will be copied." Within AWS that is merely slow. Outside AWS it is not a path at all.

GCP snapshots have a storage location fixed at creation. Google is explicit that "you can't change the storage location of an existing standard snapshot", and regionally scoped snapshots exist specifically to "control the regions that you can restore snapshots in." The feature is designed to constrain where a restore can happen.

A RunPod network volume binds you to one datacenter. Their docs state that "attaching a single network volume constrains worker deployments to that volume's datacenter, which may limit GPU availability and reduce failover options", and that "data does not sync automatically between volumes" — to get the same data elsewhere "you must manually copy data using the S3-compatible API or runpodctl." We wrote about that specific lock-in shape in the RunPod network volume alternative.

A Vast.ai volume does not even move between machines. "A volume is physically tied to the machine it was created on. It can only be attached to instances running on the same physical machine. It cannot be moved or attached to instances on other machines."

Read those together and the conclusion is structural, not incidental. Persistence products are built to make your state survive a restart in place. Portability is a different feature, and no provider has much incentive to ship it.

What actually has to be captured

The second mistake, after reaching for the provider's snapshot, is capturing the wrong layer. A working GPU environment is not a data directory. It is roughly five things, and most approaches get two or three of them.

1. Your code and working files — the obvious part, and the only part a git push covers.
2. Model weights and caches — the expensive part. Llama 3.1 70B Instruct is about 139 GB of safetensors across 30 shards by Hugging Face's own file listing. Re-downloading that is not a minor step; it is most of your restore time.
3. The installed environment — the virtualenv or conda env, the exact package versions, the compiled extensions that took twenty minutes to build.
4. System-level state — apt packages, cloned repos at specific commits, custom nodes, config files, the CUDA toolkit, things you installed at 2am and did not write down.
5. Shell and service state — what was running, and where you left off.

The common approaches each miss a chunk of that:

The Docker row is worth spelling out because docker commit is the most common DIY answer and it fails quietly. Docker's own reference says it directly: "Commits do not include any data contained in mounted volumes." If your models live on a mounted volume — which is exactly where people put them, because they are large — a committed image looks complete and restores without them. The same page notes that "by default, the container being committed and its processes will be paused while the image is committed": you are serialising filesystem layers, not a running program.

We covered the volume-versus-full-filesystem distinction in more depth in what a volume snapshot misses.

The driver trap that breaks the restore

Here is the failure that surprises people, because it happens after the copy succeeds.

You capture a working box, move it to another provider, restore every byte faithfully, and CUDA does not initialise. Nothing was lost in transit. The problem is that a piece of what you captured was compiled against the driver on the old host, and the new host has a different one.

NVIDIA's compatibility rules define the safe envelope. Minor version compatibility means "applications compiled with a CUDA Toolkit release from within a CUDA major release family can run, with limited feature-set, on systems having at least the minimum required driver version". The floors are concrete: CUDA 11.x needs driver 450 or newer, CUDA 12.x needs 525 or newer, CUDA 13.x needs 580 or newer. Within a major family you have real latitude. Across one, you do not — unless you bring the cuda-compat forward-compatibility package, which "allows applications built with a newer toolkit to run on older base drivers across major release families, subject to platform and GPU support."

Containers dodge this because the driver's user-space libraries are supplied by the host at runtime rather than baked into the image — the NVIDIA Container Toolkit's NVIDIA_DRIVER_CAPABILITIES variable "controls which driver libraries and binaries are mounted inside the container". A raw filesystem copy has no such indirection. Whatever the old host's driver put there comes along, and now it disagrees with the new card.

This is not theoretical for us. Our ComfyUI capture deliberately excludes the Python virtualenv and rebuilds it on restore, precisely because we hit this: a venv captured on a box with a newer CUDA build broke when restored onto a host with a lower driver. The models, workflows and outputs — the expensive, irreplaceable parts — come across. The venv, which is cheap to rebuild and dangerous to move, does not.

That is the general rule, and it is the most useful thing in this post: capture what is expensive and portable; rebuild what is cheap and host-coupled. Weights, datasets, configs, code and custom nodes travel. Compiled-against-this-driver artifacts should be reconstructed on arrival.

What about checkpointing the running process?

If you want the box to come back mid-computation rather than mid-configuration, you are asking for process checkpointing, and the honest status is that it is real but narrow.

CRIU cannot do it alone. NVIDIA's own tool says why: "NVIDIA GPUs provide functionality beyond that of a standard Linux kernel, and thus CRIU is not able to manage them. cuda-checkpoint adds this capability, and can therefore be used with CRIU to checkpoint and restore a CUDA application." The tool "can be used to transparently checkpoint and restore CUDA state within a running Linux process" and "supports display driver version 550 and higher."

The limits are documented and they matter: it "does not support UVM memory or IPC memory created with cuMemExportToShareableHandle()", it "waits for already-submitted CUDA work to finish before completing a checkpoint," and for multi-process jobs it "must be invoked on the processes in a job sequentially." Third-party benchmarks report failures on larger models, though that comes from a vendor comparing itself favourably, so treat it as a caution rather than a measurement.

Practical read: for the overwhelming majority of migrations you do not want process checkpointing. You want the environment back, and you restart the job. Application-level checkpoints — the ones your training loop already writes — are more reliable than freezing GPU memory, which is the same conclusion we reached in resuming training after an interruption.

The egress bill, and the one place it is waived

Moving a captured environment means moving real data — a couple of hundred gigabytes is unremarkable once weights are involved — so egress is a line item, not a rounding error.

AWS gives "100 GB of free data transfer out to the internet free each month, aggregated across all AWS Services and Regions" (excluding China and GovCloud), and charges per-GB beyond it. Check the live rate rather than trusting any figure in a blog post, this one included.

The genuinely useful exception: AWS waives egress entirely if you are leaving. Their announcement states that "starting today, we're waiving data transfer out to the internet (DTO) charges when you want to move outside of AWS", which "follows the direction set by the European Data Act and is available to all AWS customers around the world and from any AWS Region." A later update on the same page removed the previous requirement to have more than 100 GB of transfer to qualify, and gives eligible customers "90 days to complete their move off AWS." If you are migrating off AWS in earnest, apply for it rather than paying.

The sequence that works

Putting it together, in order:

1. Inventory before you capture. List what is expensive (weights, datasets, custom nodes at specific commits) and what is host-coupled (venvs with compiled extensions, driver-linked binaries). The first list travels; the second gets rebuilt.
2. Capture the filesystem, not a mount. If your capture cannot see outside a volume, it will miss the system-level state that makes the box work.
3. Write down the target's constraints first. GPU model, driver version, CUDA major family. If the new host is on an older driver than your toolkit needs, you need cuda-compat or a rebuild — find that out before the transfer, not after.
4. Move the data with egress in mind. Check whether a switching waiver applies. Compress; do not move caches you can regenerate.
5. Restore, then rebuild the host-coupled layer. Recreate the virtualenv, recompile extensions, let the package manager resolve against the driver that is actually present.
6. Verify with a real workload, not nvidia-smi. The card being visible proves very little. Run the thing you actually run.
7. Only then tear down the source. Keep the old box until the new one has done real work. The cost of a few extra hours is trivial against discovering on Monday that something did not come across.

Where we sit in this

We build this because steps 1 through 6 are too much friction for something people want to do routinely. Our capture takes the box's filesystem rather than a mounted volume, stores it in managed storage with no per-GB or per-snapshot charge, and restores it onto whichever provider you pick next — the resume path is deliberately provider-agnostic rather than a return ticket to where you started. And, as above, it is opinionated about the driver trap: the parts welded to the old host's CUDA build get rebuilt rather than blindly copied, because we shipped the version that copied them and watched it break.

If you want the conceptual version of this rather than the mechanics, moving a GPU workload to another cloud provider covers what has to come with the box, and pausing a GPU instance without losing your work covers the same machinery pointed at stopping rather than moving.

FAQ

Can I move a snapshot from one cloud provider to another? Not using the provider's own snapshot. AWS snapshots require an explicit in-AWS copy to change region, GCP fixes a snapshot's storage location at creation, and Vast.ai volumes cannot leave the physical machine they were made on. Cross-cloud migration needs a capture of the filesystem that you control, stored somewhere neutral.

Does a volume snapshot capture my whole GPU environment? No. It captures what is on the volume. Your CUDA toolkit, system packages, cloned repos and installed environment usually live on the root filesystem, outside the mount, so a volume restore leaves you rebuilding the box around your data.

Why does my restored environment fail with CUDA errors on the new provider? Almost always a driver mismatch. Something you captured was built against the old host's driver. CUDA 12.x needs driver 525 or newer and CUDA 13.x needs 580 or newer, so a newer toolkit on an older host driver fails unless you use the cuda-compat forward-compatibility package. The robust fix is to rebuild the compiled layer on arrival.

Is docker commit good enough to migrate a GPU box? Only if nothing you need is on a mounted volume, because Docker's docs state commits "do not include any data contained in mounted volumes." Since model weights are usually exactly what gets mounted, a committed image often restores looking complete and missing the expensive part.

How much does it cost to move data between clouds? Egress dominates. AWS includes 100 GB per month free and bills per-GB after that — but waives data-transfer-out charges entirely for customers moving off AWS, citing the European Data Act, with 90 days to complete the move. Check current rates directly; they change.

Can I migrate mid-training without restarting the job? Technically possible with NVIDIA's cuda-checkpoint alongside CRIU on driver 550 or newer, but it does not support UVM or certain IPC memory and handles multi-process jobs sequentially. For most people, restoring the environment and resuming from an application-level checkpoint is more reliable.

About the author

I am Ansh Saxena, working on Aquanode. I spend my time on the layer underneath the models — capturing GPU box state and moving it between providers for people who rent one card at a time and hop between clouds to chase price or availability. Everything above is either quoted from a vendor's own documentation or something we broke and fixed ourselves.

Sources

• Copy an Amazon EBS snapshot (AWS Documentation)
• About disk snapshots (Google Cloud Documentation)
• Network volumes (RunPod Documentation)
• Volumes (Vast.ai Documentation)
• docker container commit (Docker Documentation)
• CUDA Compatibility (NVIDIA Documentation)
• Specialized configurations with Docker (NVIDIA Container Toolkit)
• NVIDIA/cuda-checkpoint (GitHub)
• Amazon EC2 On-Demand Pricing (AWS)
• Free data transfer out to internet when moving out of AWS (AWS News Blog)
• meta-llama/Llama-3.1-70B-Instruct model files (Hugging Face)