How do you design high availability and disaster recovery for production ML inference services?

High availability (HA) keeps a production inference service usable when individual components fail; disaster recovery (DR) restores the complete serving capability after a larger outage. For ML, recovery must restore not just healthy HTTP containers but also compatible model weights, preprocessing, feature sources, routing, versioned configuration and valid decisions. A region that returns HTTP 200 with stale or incompatible features is not recovered.

Curriculum boundary: Question 126 planned capacity and autoscaling; Question 127 protected overloaded services with admission control and backpressure; Question 128 chose safe *per-request* fallback policies. Question 129 addresses infrastructure fault domains, coordinated service restoration, recovery-time/recovery-point objectives, cross-region artifact and feature dependencies, and disaster drills. It is not a repeat of Question 122's bad-release rollback.

Beginner-friendly incident: a delivery-risk model during a zone outage

A delivery company scores checkout orders through model late-risk/r28. All six inference replicas are accidentally scheduled on nodes in the same availability zone, and each reads features from a single zonal cache. When that zone fails, every replica and its feature source disappear. The global load balancer still has a hostname, but there are no valid predictions.

A more resilient design separates failure domains:

• Ingress and detection: A health-aware router only sends requests to an independently usable serving location.
• Compute: Distribute warm replicas across multiple nodes and zones; ensure surviving zones have enough compute, GPU quotas and model memory.
• Model and code: Keep the exact signed or checksummed deployment artifacts and feature schema available in each recovery location.
• Feature dependencies: Replicate or recreate required online features with known freshness/consistency rules, not merely model weights.
• State and decisions: Identify which records must survive: request IDs, assignments, inference logs, batch outputs, stream offsets and side-effect deduplication keys.
• Recovery control: Have one authorized owner or automation for traffic switching, failback, reconciliation and communication.

A restarted model Pod does not fix an unavailable feature store, and a successful regional routing change does not prove equivalent model behavior.

Recovery objectives, clearly distinguished

• RTO (recovery time objective): The maximum *targeted* outage duration until a minimum acceptable service is restored. Example: target 10 minutes. It is measured from the defined failure/incident start to externally verified restored service; detection and traffic convergence count.
• RPO (recovery point objective): The maximum *targeted* data loss expressed as time. Example: 2 minutes of unreplicated decision logs or state may be lost. It does not mean the model is allowed to use two-minute-old features: feature freshness is a separate correctness budget.
• Inference-specific recovery contract: Required prediction version, minimum available request capacity, input/feature validity, allowed degraded decisions, and acceptable p95/p99. A DR switch that misses these constraints has not fulfilled the business objective.
• MTTR: Observed mean time to recovery across incidents; it is a measurement, not a substitute for the defined RTO.
• RTO/RPO selection: Base targets on the business cost of missed decisions, regulatory or privacy obligations, risk severity, system dependencies and realistic recovery economics.

Important: Stateless inference replicas may have little user-state RPO, but associated feature stores, prediction audit logs, experiment assignment, stream checkpoints and business side effects can be stateful. Set RPO per component and identify the end-to-end limiting dependency.

HA within one region: first remove obvious single points of failure

1. Deploy at least two ready replicas across distinct nodes. For zonal resilience, distribute them across multiple availability zones.
2. Use topology spread constraints or suitable anti-affinity, but check the actual Pod distribution after autoscaling and rollouts; scheduler constraints are not a capacity guarantee.
3. Keep a healthy ingress path, multiple backend targets, fault-tolerant secrets/config retrieval and adequate DNS/routing behavior.
4. Verify the feature store, datastore, registry/artifact fetch and networking are not hidden single-zone dependencies.
5. Use startup/readiness probes that confirm the right model revision can actually serve a valid request. Avoid a single failing shared dependency causing all regions to thrash.
6. Set Pod disruption budgets for voluntary disruptions, but do not claim a PDB prevents involuntary node/zone loss; it only constrains eligible voluntary evictions.
7. Test a node failure and then a whole-zone failure; measure remaining RPS, p95/p99 and invalid-prediction counts, not just ready-Pod count.

Kubernetes documents topology spread across failure domains, including the risk that strict DoNotSchedule constraints cause Pods to remain pending when another zone has no eligible capacity.

Multi-region DR choices and cost trade-offs

• Backup/restore: Store immutable artifacts, manifests and consistent feature/data backups in another region; reconstruct compute when needed. Lowest steady-state compute cost, generally longest recovery.
• Pilot light: Maintain core configuration/data and minimum critical resources in a second region; scale out on demand. Faster than a full restore, but capacity, quotas and image/model pull time remain risks.
• Warm standby: Keep a smaller, functioning deployment with verified model and feature dependencies. Expand it and move traffic during an outage. Better RTO, recurring idle expense.
• Active-active: Multiple regions serve real traffic continuously. Can reduce failover time, but requires more complex state synchronization, entity routing, duplicated costs, consistency and cross-region observability.
• Provider caveat: Multi-zone or even multi-region failover is *not* always automatic for managed model endpoints. Check each service's scope and guarantees. Google Cloud's DR documentation notes that certain managed online-prediction resources are regional and do not automatically redirect to another region after regional failure.

Select by business RTO/RPO; *two regions on paper* without artifacts, feature data or quota in both is not working DR.

Numerical example 1: surviving a zone failure

Assume a model needs 180 RPS at p95 ≤ 250 ms. One warm replica has been load-tested for 40 RPS at the SLO; plan at 75% of that capacity, or 30 RPS per replica.

• Baseline required capacity = ceil(180/30) = 6 ready replicas.
• If all six are evenly spread across three zones (2/2/2), losing one zone leaves 4 replicas × 30 = 120 RPS planned safe capacity, a 60 RPS shortfall.
• To survive *any one of three equal-sized zones* with all traffic, plan 3 replicas per zone, or 9 ready replicas overall. After losing a zone, 6 remain and supply 6×30 = 180 RPS.
• This is capacity headroom for N−1 zone failure. It does not include model rollout surge, uneven traffic, other dependencies or regional incident effects.
• At a purely illustrative $0.50 per replica-hour, the 3 extra warm replicas cost 3×24×30×0.50 = $1,080 in a 30-day period, excluding acceleration, network and storage charges.

Engineering point: three-zone distribution improves fault isolation; enough surviving capacity is a separate requirement. Reserving GPU stock and upstream feature-store capacity matters as much as adding Pods.

Numerical example 2: RTO includes every step

A company commits to an RTO of 10 minutes. A regional fault occurs and the team measures:

• Detection and incident decision: 2 minutes.
• Router change and propagation: 1 minute.
• Expanding standby compute and loading the model: 4 minutes.
• Feature readiness, smoke tests and externally observed confirmation: 2 minutes.
• Total observed recovery time = 2+1+4+2 = 9 minutes, meeting the 10-minute target in this one drill with 1 minute of margin.

If the standby image registry is unavailable and model load takes 7 rather than 4 minutes, total rises to 12 minutes, missing the RTO. Keep local replicated model artifacts, warm capacity and regular drills where the objective demands them. This calculation is illustrative and not a universal cloud SLA.

Numerical example 3: RPO versus feature freshness

Suppose a feature table replicates its latest committed update to a standby region every 3 minutes. A DR plan requires RPO ≤ 5 minutes for recoverable persisted feature updates. The last replicated update is 12:00 when primary fails at 12:04:

• Replication age at failure = 4 minutes; this may fit the 5-minute *data-loss* window, assuming consistent recoverable snapshot semantics.
• But a real-time traffic model requires a feature age ≤ 60 seconds. A traffic value from 12:00 is 240 seconds old at 12:04: it is 180 seconds beyond the prediction freshness allowance.
• Therefore RPO can pass while inference correctness fails. Do not route standard predictions until valid fresh features exist; use a separately approved fallback (Question 128) or fail clearly.
• Also assess event ordering and duplicate updates when replication resumes. A timestamp alone does not prove complete, consistent feature state.

The same issue arises if model code and feature schema are replicated at different times: restore a *compatible release bundle*, not arbitrary newest components.

DR release identity and consistency

• Replicate immutable model digests, image digests, tokenizer/preprocessor dependencies and contracts to the recovery region *before* promoting a release as DR-ready.
• Store a tested deployment manifest identifying model, container, schema, feature source and operational policy. A mutable production alias may resolve differently across regions.
• Treat each environment's release readiness separately. After a candidate is promoted in primary, confirm the standby actually has the compatible artifacts and can load them; never assume the registry's metadata implies the bytes are locally available.
• Maintain recoverable feature snapshots and state with compatible schemas. Verify permissions, encryption keys, secrets and regional service quotas.
• Use idempotent request/action IDs to reconcile retries after a region switch. Old in-flight requests may finish after clients have retried in the recovery region.
• For auditability, include inference region and exact model/feature release in trace records; retained logs must follow privacy/security policy.

Safe failover and failback runbook

1. Declare incident and scope: Determine zone, regional, dependency or data corruption failure; distinguish local deployment bug from regional outage.
2. Check recovery readiness: Validate candidate region's model digest, features, secrets, quotas, routing health and predicted demand. Avoid blindly switching into an already overloaded region.
3. Fence unsafe writers/side effects: Coordinate active writers, consumer leases and side effects where split-brain is possible; define safe handling of ambiguous in-flight business transactions.
4. Shift traffic deliberately: Route to healthy targets through a configured global load balancer/accelerator or approved DNS procedure. DNS TTLs and cached connections make convergence noninstantaneous.
5. Measure externally: Run authorized real prediction probes, verify freshness, p95/p99, error rate, actual request share and capacity, then timestamp recovery.
6. Reconcile state: Repair missing logs, prediction IDs, late events, duplicate actions and uncommitted batch partitions; respect RPO and notify consumers when outputs may be incomplete.
7. Fail back only after stability: Restore source data and artifacts, compare model/feature parity, drain in-flight work and transition traffic during an approved maintenance window. Failback can cause a second outage if rushed.
8. Capture evidence: Record timeline, observed RTO/RPO, invalid decisions, failed dependencies and actions for the next recovery drill.

Automated failover needs strong health checks and anti-flapping rules. Partial regional failures can be worse than clean outages; false health positives can switch traffic to a bad location, while correlated dependencies can make both regions unhealthy.

Mode-specific recovery

Online synchronous serving: Focus on routing healthy endpoints, warm model replicas, strict request deadlines and current enough features. Do not mistake server availability for valid model decisions.

Asynchronous/batch scoring: Preserve job/run identities, source snapshot, model release, partition receipts, output manifest and idempotent commits. Re-run incomplete logical intervals from a stable checkpoint rather than exposing duplicate or partial predictions.

Streaming inference: Preserve broker offsets, state snapshots/checkpoints, operator/state schema compatibility and sink idempotence. Recovering from an old checkpoint can replay events and repeat external side effects; coordinate consumer ownership and output deduplication. Regional event replication has its own lag and data-loss trade-offs.

Hybrid serving: The online service, batch scorer and streaming feature updates may recover at different times. Define which outputs can be considered authoritative and document the allowed degraded state.

Practical infrastructure snippet (illustrative Kubernetes topology)

YAML
spec:
  replicas: 9
  template:
    metadata:
      labels:
        app: risk-inference
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: risk-inference

This is a workload Pod-template fragment, not a complete Deployment or an SLA guarantee. It needs eligible nodes in the zones, suitable compute quota, readiness probes, anti-disruption policy and a realistic ingress distribution. Strict spreading can leave Pods pending in capacity-constrained zones. Verify that all zones can schedule the required model image and GPU resources.

Monitoring and failure drill checklist

• Availability: Success and latency per region/zone, external synthetic predictions, real model-version response checks and user-visible errors.
• Recovery targets: Actual incident start, detection, failover start, routing convergence, first valid response, RTO achievement and RPO age by state store.
• Capacity: Ready vs desired replicas, zone placement, warm model memory, accelerator quotas, headroom after N−1 failure, feature-store load and saturation.
• Data correctness: Feature age, schema/version mismatch, replication lag, dropped/replayed events, duplicate actions and output-manifest completeness.
• Control plane: Global route weights, health-check confidence, controller state, regional artifact availability, IAM/secret/key access, and runbook age.
• Cost: Warm standby hours, duplicate storage/replication, cross-region transfer and cost of overprovisioned N−1 protection.

Run controlled drills: kill one inference Pod, disable a zone's targets, make a critical feature store unavailable, fail a region's ingress, restore from a snapshot and verify a failback. Coordinate production-impacting tests with incident owners and stop conditions. Measure actual customer-visible recovery, not just whether a cluster created Pods.

Common mistakes and practical trade-offs

• Mistake: Distributing Pods across zones but putting the feature cache or secret store in a single zone.
• Mistake: Assuming a Kubernetes disruption budget prevents involuntary zonal outages.
• Mistake: Saying “RTO 5 minutes” without including fault detection, model warmup and traffic propagation.
• Mistake: Confusing feature freshness SLO with backup/replication RPO.
• Mistake: Keeping a standby region whose models, credentials, quotas or schemas drift from the primary.
• Mistake: Treating DNS or global routing updates as instantaneous and ignoring clients pinned to old connections.
• Mistake: Switching to a region that has no tested capacity for the full production traffic.
• Mistake: Allowing both regions to process irreversible business writes without split-brain controls.
• Mistake: Assuming a healthy Kubernetes Pod means valid ML predictions.
• Mistake: Never exercising failback, data reconciliation or disaster recovery after a model update.
• Trade-off: Active-active often reduces detection-to-routing time but increases complexity and expense; warm standby can meet many RTOs with less cost, provided periodic failover drills confirm usable capacity.
• Trade-off: Stricter topology spread improves fault isolation but can impede scheduling when healthy zones lack GPU nodes.
• Trade-off: Synchronous cross-region state replication can improve consistency at the cost of write latency and availability; asynchronous replication preserves latency but allows lag/data-loss windows.

Takeaway: A resilient ML deployment restores a compatible, sufficiently fresh, adequately provisioned prediction service, not merely a container or DNS record. Define RTO/RPO separately from prediction freshness, eliminate hidden single points of failure, replicate immutable releases and dependent state, and repeatedly demonstrate recovery and failback within measurable objectives.