Migrate a GPU Workload Between Two Clouds Without Downtime
A step by step guide to migrating a running GPU workload between two clouds without downtime, covering parallel running, traffic shifting, data sync, and rollback.
Cross-cloud migration sounds risky because it usually is when done as a single cutover. The safe approach treats migration as a gradual traffic shift between two environments that run in parallel for a while. This tutorial covers how to move a GPU-backed inference or training-serving workload from a source cloud to a target cloud without downtime, the pitfalls that cause silent failures, and how to roll back if the new environment misbehaves.
Why Teams Migrate
The usual drivers are price and availability. A neocloud may offer the same GPU class at a lower hourly rate, a hyperscaler may run out of a specific accelerator in your region, or a contract may be ending. Whatever the reason, the workload itself rarely changes. Your job is to reproduce the runtime faithfully on the target and move traffic across without users noticing.
Establish Parity First
Before any traffic moves, the target environment must be a faithful copy. Differences in driver version, GPU model, or library builds can change numerical output or latency in ways that surprise you in production.
- Match the GPU model and memory size, or document any intentional change.
- Pin driver and runtime versions, including the inference server and framework builds.
- Reproduce environment variables, model weights, and configuration exactly.
- Confirm region and networking so latency to your users stays comparable.
Run a smoke test on the target that compares outputs against the source for a fixed set of inputs. Any divergence here is far cheaper to fix now than after traffic arrives.
Synchronize Data and State
The hardest part of zero-downtime migration is shared state. Stateless inference is easy because each request is independent. Anything that holds state needs a plan.
- Model artifacts: copy weights and caches to the target ahead of time, then verify checksums.
- Vector stores or databases: replicate to the target and keep them in sync until cutover, or point both environments at a shared managed store during the transition.
- Caches: accept a cold cache on the target initially, or warm it before sending real traffic.
Decide early whether the two environments can share a backing store or whether you need live replication. Sharing is simpler but adds cross-cloud network latency, so measure it.
Shift Traffic Gradually
With both environments healthy, move traffic in small increments rather than all at once. A load balancer or DNS-based router in front of both clouds lets you control the split.
| Stage | Target share | What to watch |
|---|---|---|
| Canary | Small slice | Error rate, latency, output parity |
| Ramp | Growing share | Saturation, queue depth, cost |
| Majority | Most traffic | Stability under real load |
| Cutover | All traffic | Source idle, ready to roll back |
At each stage, hold long enough to see real patterns, including peak load. If the target shows higher error rates or latency, pause and investigate before ramping further.
Keep a Rollback Path
Until you fully decommission the source, keep it warm and ready. Rolling back should be as simple as shifting the traffic split back toward the source. This is why a gradual approach beats a hard cutover: every stage is reversible. Only retire the source once the target has run the full traffic load through at least one peak cycle without issues.
Mistakes That Cause Outages
- Cutting over all traffic at once with no canary stage.
- Forgetting cross-cloud egress costs, which can dominate during data sync.
- Assuming identical GPU model means identical behavior when driver versions differ.
- Decommissioning the source before the target has survived peak load.
- Ignoring latency added by a shared backing store across two clouds.
Egress in particular catches teams off guard. Moving large model artifacts and replicating data between providers can generate a real bill, so budget for it and prefer one-time transfers over continuous cross-cloud chatter.
Watch Observability During the Shift
A traffic shift you cannot see is a gamble. Before the first request moves, make sure both environments emit comparable telemetry so you can compare them side by side. The metrics that matter most during migration are error rate, latency at the tail rather than just the average, and output correctness on a sampled basis.
- Error rate: a rise on the target is the clearest signal to pause the ramp.
- Tail latency: averages hide problems; the slowest requests reveal saturation and cold caches.
- Output parity: sample real responses from both clouds and compare, since a subtle numerical drift may not raise errors at all.
- Saturation: queue depth and GPU utilization tell you whether the target is sized correctly for its share.
Keep dashboards that show source and target next to each other for the duration of the migration. The goal is to make divergence obvious at a glance so you react in minutes, not after a user complains.
Handle Capacity and Quota Ahead of Time
A migration can stall not because the plan is wrong but because the target cloud will not give you the GPUs when you need them. Specific accelerators can be scarce in popular regions, and new accounts often start with low quotas. Request the capacity and raise the quotas well before the cutover window, and validate that you can actually launch the full target footprint, not just a single test instance. Discovering a capacity ceiling halfway through a ramp is a painful place to be, because the source may already be partly drained.
Right-Size and Re-Optimize on the Target
Faithful parity gets you a safe migration, but the destination cloud is also a chance to improve. Once traffic is stable, revisit instance sizing on the target with fresh utilization data. The mix of reserved, on-demand, and spot that was optimal on the old cloud may not be optimal on the new one, since rates and availability differ. Treat post-migration tuning as a deliberate second phase: migrate first to prove stability, then optimize to capture the cost advantage that justified the move in the first place. Bundling the two phases together makes it hard to tell whether a problem came from the move or the optimization.
A zero-downtime cross-cloud migration is mostly discipline, not magic. Build a faithful target, synchronize state deliberately, shift traffic in measured increments, and keep rollback one step away at all times. When you reach full cutover, you will have proven the new environment under real load rather than hoping it works. Only then should you turn off the old cloud and capture the savings that prompted the move.