How do you design chaos experiments and disaster-recovery procedures for distributed LLM inference systems?

Chaos engineering for distributed LLM inference deliberately injects a *bounded, observable failure* to verify that routing, GPU worker replacement, queueing controls and recovery procedures actually preserve user-facing service objectives. Disaster recovery (DR) expands the scope to catastrophic node, zone, region or control-plane failures and proves that traffic, configurations, model weights and essential state can be restored to an agreed service level.

The objective is not to break GPUs for its own sake. Each experiment starts with an explicit hypothesis and a stop condition, tests a single fault domain, measures real request outcomes, and produces a change to improve the system.

Unlike Question 255 (designing failover), Question 263 (graceful rollout draining), and Question 270 (triaging individual GPU/CUDA incidents), this lesson teaches how to validate resilience systematically with controlled fault injection and complete DR drills.

1. Establish the steady state and blast radius

Before injecting failures, record:

• Customer SLIs: original request success rate; p95/p99 time to first token (TTFT), inter-token latency (ITL/TPOT), full-stream completion, error/rejection rates and SLO-compliant goodput.
• Dependency health: ready-versus-desired replicas, prefill/decode worker counts, KV-cache occupancy, network transfer wait, router health, GPU hardware status and load-balancer endpoint propagation.
• Recovery SLIs: time to detect the incident, time to stop routing to the failed worker, time to ready replacement capacity, and time for p95 TTFT/ITL to return below SLO limits.
• Workload: representative, privacy-safe production-like prompt/output length mix, open-loop arrival schedule, streaming, burstiness and tenant priorities.

Pick a steady-state hypothesis such as: “With one decode worker lost, 99% of original user requests still complete within our target latency, no active streams silently disappear, and full rated capacity is restored within five minutes.”

Start in an isolated staging cluster with synthetic or approved redacted prompts, then move toward a narrowly scoped production experiment only with change approval, on-call coverage, monitoring and an immediate abort switch.

2. Define what to break and what it teaches

Worker-process kill

• Terminate one noncritical inference engine process deliberately; observe how readiness, routing, active streams and replacement behave.
• A crash is *not* a graceful drain. If in-flight streams cannot resume, count the lost user requests as failures rather than hiding them in retry totals.

GPU device or node loss

• Simulate node outage / eviction / device unavailability in a safe test environment. Check GPU quotas, alternative placement, tensor-parallel group rebuilding and cold-start overhead.
• A single GPU failure can invalidate an entire multi-GPU tensor-parallel replica, so the fault domain is larger than “one GPU worth of requests.”

Prefill or decode worker loss

• In disaggregated inference, failure of a prefill worker may cause a different failure shape from decode worker loss. The former can stall new prefills, while the latter can interrupt active output streams and strand KV transfer state.
• Verify the router does not schedule new requests to dead workers and that stale KV handles are reclaimed.

Network degradation/partition

• Inject latency, packet loss, disconnects or partitions between gateway/router, prefill/decode workers and KV-transfer services. Test deadlines, circuit breakers, transfer cleanup and backpressure.
• Example tool: LitmusChaos pod-network-latency, which can target specific pods/destinations and delay traffic for a bounded time. Validate supported experiment options against the deployed version.

Model storage or registry outage

• Remove access to remote checkpoints or model registry for a bounded test; verify existing ready replicas keep serving while new cold replicas might fail to start.
• Test availability of verified checkpoint copies, access controls, image caches and model-version pinning.

Control-plane or region disruption

• Test what happens when autoscaling/control-plane updates or a full availability zone becomes unavailable. Existing inference traffic may survive some control-plane failures but not all data-path failures.
• For region-level DR, route only to an explicitly approved region with compliant data-residency and model/version policies.

Overload combined with a fault

• Combine a controlled worker loss with a representative arrival burst *only after* passing one-fault experiments. This tests remaining capacity, retry amplification and queue protection.

3. Beginner-friendly numerical example

Suppose a distributed LLM deployment uses four independent serving replicas, each able to sustain 6 SLO-compliant requests/s on the same measured workload.

• Total safe modeled capacity = 4 × 6 = 24 requests/s.
• Current offered demand = 18 requests/s.
• An experiment kills one replica. Remaining nominal safe capacity = 3 × 6 = 18 requests/s.
• At the *average* incoming rate, the remaining fleet has zero modeled spare headroom. Any short surge or longer prompt distribution may cause SLO failure.
• A second unexpected loss leaves 2 × 6 = 12 requests/s. The excess demand is 18 − 12 = 6 requests/s.
• An unbounded queue would accumulate approximately 6 × 60 = 360 requests after one minute of sustained deficit, ignoring cancellations/timeouts/rejections.
• If a replacement becomes inference-ready after 180 seconds, the naive unbounded backlog would have grown by roughly 6 × 180 = 1,080 requests during the two-replica shortfall.

Correct response: cap work admitted by token budgets and user deadlines, shed/defer excess traffic, bring up compatible ready replicas, and include rejections/stream aborts in success metrics. A queue that absorbs 1,080 requests long after their deadlines is *not* resilience.

The figures are illustrative. The independent “six good requests/s per replica” assumption can fail when shared network, storage, router and disaggregated worker dependencies saturate.

4. Hypothesis, procedure and stop conditions

A reusable experiment specification should record:

1. Experiment ID and owner: affected model/version, namespace, GPU pool, approved time window and on-call operator.
2. Steady-state hypothesis: specific SLO thresholds, error tolerance and recovery-time objective.
3. Fault: one target process/pod/GPU or narrow network rule, duration, trigger method and reversibility.
4. Blast-radius limit: test namespace, one eligible replica or a fixed small percentage of capacity, and excluded high-priority tenants.
5. Baseline: normal traffic rate, prompt mix, warm-cache state, version, ready GPU count and pre-fault metrics.
6. Abort conditions: for example sustained >1% stream failures, p95 TTFT above a critical threshold, failed rollback, or any unexpected tenant/security impact. Set thresholds from approved SLOs, not generic defaults.
7. Observation window: pre-fault, fault-injection and post-recovery metrics plus traces and incident timeline.
8. Rollback action: remove injection rule, uncordon eligible healthy nodes, restore routing and validate request completion.
9. Outcome: hypothesis passed/failed/inconclusive, supporting graphs, follow-up changes and retest owner.

If telemetry, abort tooling or healthy spare capacity is missing, do not proceed with production chaos injection.

5. Practical Kubernetes fault-injection approach

LitmusChaos supports container kill, pod delete, pod network loss/latency/partition, CPU stress and node-level experiments. Its network-latency experiment describes a controlled packet-delay injection at targeted pods/destinations.

A *safe conceptual sequence* (not a command to execute on production automatically):

• Identify the dedicated test namespace and one disposable inference worker.
• Run a synthetic streaming client continuously and record TTFT, ITL, full-response success and error codes.
• Inject a bounded single-worker termination or 200-ms network delay, with predeclared stop and time limits.
• Observe endpoint removal, failure detection, request results, standby capacity, GPU provisioning, KV cleanup and latency recovery.
• Remove the fault and verify the system returns to steady-state performance.

Kubernetes PodDisruptionBudgets: PDBs constrain voluntary Eviction API disruptions but cannot prevent involuntary hardware failures; ordinary direct pod deletion and workload rolling-update behavior require separate protections. A PDB alone is therefore not a chaos safety switch. Use explicit target allowlists and permissions.

6. Recovery objectives: RTO and RPO

• RTO (recovery time objective): maximum acceptable elapsed time from a qualifying outage until the defined service is restored to its agreed state. Specify whether this means *first response possible* or *normal SLO performance restored*.
• RPO (recovery point objective): maximum acceptable age/amount of persisted state lost after recovery.
• In a stateless inference-only API, model weights/configuration usually come from versioned storage; RPO for user state may be not applicable. In-flight KV caches are ephemeral and normally cannot be reconstructed token-perfectly after a worker crash.
• In an agentic or conversational product, session history, tool execution records, authorization context and conversation metadata may live outside inference GPUs. Their RPO must be defined and enforced with backups/replication, and replaying a tool action needs idempotency.
• Do not promise lossless continuation of a partially streamed generation unless the serving stack explicitly implements and tests a compatible continuation protocol.

7. A disaster-recovery runbook

1. Declare and scope the incident: identify failing region/zone, missing model replicas, storage/network dependencies and customer impact.
2. Freeze risky automation: pause deployments that could amplify the incident; rate-limit retrying clients and cancel impossible deadlines.
3. Select approved recovery region/zone: confirm GPU inventory, topology, model availability, data residency and tenant permissions.
4. Restore control and data dependencies: image registry, version-pinned model checkpoints, configuration/secrets, routing metadata, auth, observability and any required session databases.
5. Provision whole serving groups: load model weights, compile/warm runtime, validate tensor parallelism, adapters, KV allocation and readiness with a real inference probe.
6. **Shift eligible *new* traffic gradually:** apply circuit breakers and guarded traffic weights; preserve healthy remaining regions and monitor error/latency/quality changes.
7. Handle active user streams explicitly: report interrupted streams and let clients resume/retry from application state where possible, not by assuming migrated GPU KV state.
8. Verify recovered SLOs and state: measure original-request success, p95/p99 TTFT/ITL, SLO goodput, tool action integrity, auth, model checksum and backups' recovery point.
9. Return/operate in DR mode: keep production in approved fallback until stable and confirm rollback/failback readiness. Record outage and actual achieved RTO/RPO.

Important: keeping a checkpoint in storage does not guarantee a rapid RTO. A new region must also have compatible GPU capacity, sufficiently provisioned network, cached model weights and functioning dependencies.

8. Recovery experiments versus vendor-specific fast restart

NVIDIA Dynamo documents a shadow-engine recovery feature in which an initialized standby engine on the same GPU can reduce engine-process recovery time. A published demonstration (August 2026) showed 7.3 seconds to restore a second worker, versus 283 seconds for cold restart in a specific two-worker B200/GLM-5.2 experiment.

Scope of the evidence: this addresses a class of process/engine failures where the GPU remains available. It does not automatically recover from GPU destruction, node/zone loss, corrupted persistent user state or a region outage. In chaos tests, distinguish process recovery, device recovery, worker-group rebuild, and full disaster recovery rather than mixing their success criteria.

A fast process restart can reduce lost capacity without preserving in-flight stream progress; validate what happens to already active requests.

9. Example production incident

Scenario: A GPU node hosting a decode worker becomes unavailable while the router continues sending new decode assignments to stale endpoints. Meanwhile, client retries multiply incoming work. Prefill GPUs appear healthy, but KV transfers fail and TTFT spikes.

Root cause: incomplete worker-health propagation and stale routing entries, combined with retry amplification and inadequate decode headroom.

Immediate containment:

• Withdraw failed decode endpoints, bound admissions and retries, and preserve healthy worker-group capacity.
• Invalidate orphaned KV transfer handles and account for interrupted streaming requests.
• Bring up a compatible decode worker/group on available healthy GPU capacity and wait for inference readiness.

Prevention experiment: Kill exactly one test decode worker during a representative open-loop mixed workload. Validate that the router removes its endpoint within the target detection window, no new transfer is sent to it after convergence, uncompleted requests fail clearly rather than hanging, and overall service returns to its latency SLO inside RTO.

Avoid false success: a pod returning to Running state does not prove that a compatible model is loaded, the KV data path works, or previous requests were correctly accounted for.

10. Measure resilience and learn

For each experiment publish:

• Before/during/after: offered/accepted/completed RPS, original-request SLO success, errors/429/503, stream completion, p95/p99 TTFT and ITL.
• Recovery timings: detection latency, route withdrawal, GPU replacement start, model time-to-ready, and SLO restoration.
• Distributed state: KV transfer failures, orphaned buffers, failed tensor-parallel ranks, session/tool state integrity.
• Safety: whether stop conditions triggered, whether the injection was removed, and whether customer data/residency policies stayed enforced.
• Coverage: process kill, node/GPU failure, network degradation, model storage outage, prefill/decode split, and region DR as applicable.

Trade-offs:

• Larger spare GPU pools make failover faster but cost more; calculate cost per successfully SLO-conforming request during fault scenarios.
• Frequent chaos tests improve confidence but should not increase real customer risk; start small and expand only with evidence.
• Reconstructing KV and replaying prompts costs latency/tokens and may produce a different answer. Preserve application-level state and guard side effects.
• Multi-region standby may be constrained by GPU scarcity and data residency. Test the actual hardware/model deployment path, not merely DNS switching.

Interview-ready answer: “I would define the steady-state SLIs, fault hypothesis, blast radius, stop conditions and RTO/RPO before injecting anything. I'd start with controlled worker and network failures, then exercise GPU/node loss and full region recovery in dedicated environments. For LLMs I'd explicitly test in-flight stream loss, prefill/decode/KV dependencies, model warmup and retry storms. I'd measure original-request goodput and the time to restore SLOs—not just pod restarts—then turn each failed experiment into a runbook and regression test.”

References:

• Kubernetes — Disruptions and PodDisruptionBudgets
• LitmusChaos — Pod Network Latency
• NVIDIA Dynamo — Inference Architecture
• NVIDIA — Shadow Engine Recovery (2026)
• Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime (2026)