How do you design multi-region failover and capacity planning for highly available LLM inference?

Multi-region LLM inference places independent, compatible model-serving capacity in more than one geographic cloud region so requests can be served despite a region-wide outage—or routed closer to users. The crucial question is **whether the surviving region has enough *ready, SLO-compliant GPU capacity* for redirected traffic**, not whether DNS can point somewhere else. Large checkpoints, GPU scarcity, long model cold starts, KV-cache locality, streaming connections, and data-residency requirements complicate recovery beyond typical stateless web services.

This lesson builds on Question 255 (health-aware replica failover), Question 260 (GPU sizing), Question 261 (SLO-aware autoscaling), Question 262 (cold-start prewarming), and Question 271 (chaos and DR experiments). It specifically covers geographic fault domains, active-active versus active-passive inference, failover capacity math, global routing, cross-region state and cost.

1. Decide if multiple regions are justified

• A multi-availability-zone design within one region may already meet many reliability requirements.
• Multi-region deployment becomes appropriate when the service must survive an entire regional outage, meet geographically separated recovery or data-residency obligations, or reduce latency for users in distant locations.
• It adds extra GPU and storage spending, checkpoint replication, traffic-management complexity, data consistency concerns and more failure modes.
• Define a measurable regional outage SLO, recovery time objective (RTO), recovery point objective (RPO), and whether degraded model quality or partial traffic shedding is acceptable during recovery.
• Do not interpret “multi-region” as automatic immunity to shared control-plane, identity-provider, DNS, cloud-account, model-registry or software-version failures.

AWS Prescriptive Guidance emphasizes choosing multi-region architecture when a documented driver requires it, rather than making it a default for all workloads.

2. Four recovery patterns

Backup and restore: replicate versioned infrastructure configuration, model artifacts, databases and backups to a recovery region, but create most compute only after an outage. Lowest ongoing GPU expense; recovery may be long and depends on quota, GPU availability and weight downloads.

Pilot light: keep critical control/state dependencies running in the DR region but provision most model-serving GPUs on demand. Faster than full restoration but still requires node creation, checkpoint loading and engine warmup.

Warm standby: keep a smaller *working, healthy inference deployment* ready in the backup region. It serves little/no user traffic until a failover and then scales out. Existing warm capacity protects some urgent requests, but expanding to full capacity may take minutes.

Active-active: run user-serving GPU pools in two or more regions and distribute live traffic. Routing can shift traffic when one region fails, but surviving regions still need failover headroom or they will be overloaded.

RTO/RPO labels in vendor diagrams are architectural tendencies, not guaranteed durations. An LLM warm standby with insufficient GPU quota or a 15-minute model download can miss its recovery objective.

3. Beginner-friendly numerical capacity example

Assume a single one-GPU serving replica has a *measured, safe operating capacity* of 5 SLO-compliant requests/second for the real prompt/output-token distribution, including chosen headroom. Two regions each receive 20 requests/second during peak business hours.

Normal active-active operation:

• Region A: \ceil(20 / 5) = 4 replicas\.
• Region B: \ceil(20 / 5) = 4 replicas\.
• 8 GPUs total for 40 RPS; each region is sized for its local share.

If Region A fails:

• The surviving Region B receives 40 RPS if all traffic is eligible for redirection.
• Required replicas: \ceil(40 / 5) = 8\.
• Its existing 4 replicas can serve only 20 RPS at the planned safe rate.
• Shortfall = 20 RPS until four additional compatible GPU replicas become ready, even though global failover is functioning.

True immediate regional-outage resilience:

• Preposition 8 ready replicas in each region to handle the entire 40-RPS demand from either side.
• Fleet size: 16 one-GPU replicas, compared with 8 GPUs when only normal demand is covered.
• This is an illustrative, expensive N-region capacity design; actual replication, GPU topology and shared bottlenecks alter the count.

Warm-standby alternative: Region B keeps 2 ready spare replicas (10 RPS safe capacity) and scales to 8 on failover. With a 40-RPS surge, 30 RPS remains above immediate warm capacity. This offers some fast service but cannot promise full peak SLO compliance until more replicas are ready. In practice it may require priority tiers, queue limits and shedding.

Important: passing SLOs at 5 RPS/replica under steady load does not prove a 40-RPS redirected burst can be handled by aggregate averages alone. Verify queueing, load balancing, cache coldness, network path, tenant mix and fault-domain independence.

4. Capacity planning formula with loss of a region

For two equivalent regions with normal demand \D_A\ and \D_B\:

• Total eligible demand = \D_total = D_A + D_B\.
• Let \c\ = benchmarked safe goodput of one complete serving replica at the specified SLO.
• To survive loss of either region *immediately*, each surviving region must have at least \R = ceil(D_total / c)\ ready replicas, subject to token-mix, routing and shared-resource constraints.
• If a replica requires \g\ GPUs (e.g., four-way tensor parallelism), each region requires R × g GPUs, not R GPUs.
• If the requirements instead allow a few minutes of degraded service, you can deploy fewer warm replicas and explicitly model the recovery/scale-up interval and allowed traffic reduction.

For nonidentical GPUs or model variants, use measured per-region capacity envelopes rather than multiplying unlike replica counts. Recalculate with p95 peak load, extra burst headroom, model startup time, and separately defined N−1 replica/zone requirements.

5. Recovery-time budget: traffic switch is not capacity readiness

Break the actual failover timeline into:

1. Detect loss of user-facing model service, not just a reachable TCP socket.
2. Validate the incident to avoid flapping or falsely isolating a slow but healthy region.
3. Update health-aware global routing and wait for its effective propagation to clients/proxies.
4. Confirm the target region has compatible ready inference replicas and room for more sessions.
5. If capacity is missing, allocate GPUs, start containers, stage checkpoint shards, load weights, compile/capture kernels and warm the serving stack.
6. Verify p95 TTFT/ITL, errors, original-request success and full streaming completion at redirected load.

Illustrative recovery example: failure detection 20 s, global route convergence 15 s, replacement provisioning+model warmup 180 s. Time until full targeted capacity = about 215 seconds, assuming sequential critical-path stages. Prepositioned warm GPU capacity could eliminate the 180-second component *for the work it can already handle*. These are hypothetical times.

DNS health checks and low TTLs can assist but do not guarantee every existing client stops using an old endpoint immediately: caching resolvers, persistent HTTP connections and streaming sessions can outlive a DNS change. Global anycast or application gateways have different convergence behavior; measure client-observed recovery, not just control-plane status.

6. Region-level global traffic routing

• Route only among healthy, model-compatible and authorized endpoints. A region containing a healthy web gateway but unavailable prefill or decode GPUs is *not* fully healthy.
• Combine geographic/latency-aware routing with capacity-admission signals so a “nearest healthy” region cannot attract more token work than it can serve.
• Provide transparent region exclusion, failover weights and optional manual override; automate decisions with hysteresis to prevent traffic flapping.
• AWS Route 53 supports active-active and active-passive failover routing using health checks. A DNS policy does not itself assess model-specific TTFT, KV pressure or ready GPUs; integrate an application-level health endpoint or supported load balancer policy.
• Google Cloud documents serving in multiple regions behind global load balancing and health-based regional failover for supported deployments.
• Preserve per-tenant routing constraints; a user assigned to an EU-only residency boundary must not be redirected to an unapproved region merely because capacity exists there.

7. What state must be replicated?

Immutable or versioned artifacts:

• Model weights, tokenizer, chat templates, quantization configuration, adapters, inference engine/container manifests, safety policy and tool schema.
• Store immutable digests/checksums and verify the recovery region runs the same approved version, unless an explicitly tested fallback is allowed.
• Keep cross-region artifact copies or caches near recovery GPUs; a single-region object-store dependency makes a multi-region deployment fragile.

Persistent application state:

• Auth/session metadata, conversation turns, retrieval index or vector store, tool execution ledger, quotas/billing and any customer data needed to answer requests.
• Define consistency and RPO for each component. Async replication may lose the latest writes; active-active writes can conflict without ownership or conflict resolution.
• Use idempotency keys and transaction status for side-effecting tools, especially when a client retries after a cut-off stream.

Ephemeral GPU serving state:

• KV-cache blocks, batching queues and an in-progress decoder state are generally bound to a running worker and not replicated automatically across regions.
• A failed region's live streams normally terminate, and restarting a generation may produce different tokens or repeat side effects.
• Replaying a prompt from persisted conversation history is an application-level retry, not bit-identical token continuation.

State recovery must preserve tenant authorization, retention, encryption, data locality and model licensing.

8. Disaggregated prefill/decode across regions: a common mistake

• Prefill and decode workers may exchange large KV state over high-bandwidth, low-latency fabrics; locating a prefill GPU in one distant region and decode GPU in another can add severe transfer delay and cross-region bandwidth cost.
• Prefer regional self-contained serving cells with colocated or region-local prefill/decode/KV dependencies and local routers.
• Fail over whole compatible regional serving paths instead of trying to move live KV blocks over intercontinental links.
• Keep local GPU topology (NVLink/PCIe), KV network and storage budgets separate from the global request-routing layer.
• If the product supports explicit cross-region context migration, benchmark transfer cost, consistency and in-flight recovery; do not assume that ordinary model serving has it.

NVIDIA Dynamo describes a distributed serving architecture with routing, disaggregated inference and KV-state paths. Its ability to scale workers within a serving environment does not, by itself, provide automatic regional disaster recovery or global session-state replication.

9. Production incident: DNS switches but recovery still fails

Incident: Region A receives 70% of user traffic and Region B receives 30%. A regional outage triggers global health checks and sends the failed region's traffic to B. DNS switches within seconds, but B's p95 TTFT jumps from 800 ms to 9 seconds, and error rates surge.

Root-cause debugging:

1. Routing shows the global controller worked as intended, but Region B has only enough ready GPUs for its usual 30% share plus small spare headroom.
2. Autoscaler asks for new GPUs, but the target region lacks immediately available quota/stock and must download large checkpoints from a cross-region store.
3. Cold prefix caches lower prefill reuse, increasing actual token work versus the benchmark.
4. User retries compound the overload, and long queues retain work beyond request deadlines.

Engineering solution:

• Preposition compatible warm capacity sized to the explicit degraded-mode SLO and regional-loss requirement.
• Pre-stage checkpoints and container images in B; validate usable GPU quota and model startup p95.
• Make global routing aware of healthy ready capacity and per-tenant constraints, not merely DNS endpoint health.
• Bound admission, honor deadlines, jitter/back off retries and allow defined priority-based degradation during recovery.
• Benchmark full redirected peak traffic with representative long-context and cold-cache conditions.

Prevention: Regularly conduct a controlled entire-region traffic evacuation test, measuring routing convergence, ready-replica counts, SLO-goodput, stream aborts and actual RTO/RPO. A healthy DNS endpoint alone is not a passed disaster-recovery drill.

10. Recovery economics and failure trade-offs

• Active-active: stronger immediate regional survival and lower geographic latency, but requires duplicated GPUs and careful cross-region state synchronization.
• Warm standby: lower idle GPU spending, but the service must declare what percentage of requests it can handle before additional model replicas become ready.
• Pilot light: may reduce idle costs but depends heavily on GPU supply, checkpoint download and warmup during an incident.
• Model fallback: an approved smaller or alternate model can preserve some workflows at different cost/accuracy; only use it when contract, quality, safety and data-residency conditions permit.
• Cross-region egress: moving prompts, retrieval context, output and persistent state consumes network bandwidth and may have financial/privacy impact. Avoid large KV transfers across distant regions without measured benefit.
• Cold cache penalty: loss of regional prefix/KV reuse raises prefill work, so compute failover capacity using the expected post-disaster cache state, not only warm steady-state benchmarks.
• Total cost metric: spend per *SLO-successful user request*, including idle warm GPU capacity and replicas provisioned for regional failure.

11. A validation checklist and runbook

Before rollout:

1. Record steady-state load by region, peak total demand, measured safe goodput per compatible GPU replica, and chosen RTO/RPO.
2. Verify two-region loss-of-one-region capacity (or document the explicit temporary degraded-service contract).
3. Confirm immutable model artifacts, tokenizer/adapters, routing, auth, retrieval state and versioned configuration are available in each approved region.
4. Test routing failover, unhealthy model workers, GPU cold starts, quota limits and shared external dependency failures.
5. Test long-lived streaming sessions, idempotent retries and tool-call side effects across regional cutover.
6. Confirm telemetry distinguishes original requests, backend retries, errors, p95 TTFT/ITL, full-stream completions and SLO-goodput per region.
7. Rehearse failback gradually; do not immediately move all requests to the recovered region before it is warmed and verified.
8. Validate data-residency and security rules in routing, backups and cross-region observability.

Interview-ready answer: “I would decide whether the workload needs warm standby or active-active based on a regional-loss RTO and explicit SLOs. Then I'd size every surviving region for the total eligible failover demand using measured per-replica SLO goodput, not normal regional load. I'd keep immutable model artifacts and critical state available regionally, prewarm GPU capacity or document reduced service during scale-up, and route only to ready, compatible, authorized regions. Finally, I'd drill full-region evacuation with cold caches, live streams and retries and report actual SLO restoration time and RPO.”

Research references:

• AWS Prescriptive Guidance — Multi-Region Architecture
• AWS Well-Architected — Disaster Recovery Options
• Amazon Route 53 — Active-Active and Active-Passive Failover
• Google Cloud — Serve Traffic From Multiple Regions
• NVIDIA Dynamo — Overall Architecture