How do you diagnose and recover from GPU OOM crashes, CUDA errors, and unhealthy LLM inference workers in production?

Production inference recovery starts by identifying the failure class, isolating the affected worker or GPU, protecting active requests, and replacing capacity safely. A CUDA out-of-memory (OOM) error, an illegal-memory-access kernel failure, a failed tensor-parallel collective, and a hardware Xid alert can look similar to an end user, but they do not have the same safe recovery procedure. Treat availability, SLOs, and data integrity as first-class outcomes—not merely whether a restarted process becomes reachable.

This lesson builds on Question 269 (leaks and fragmentation), Question 263 (stream-safe draining), and Question 255 (general failover). Its focus is incident classification, node/worker health, crash containment, safe recovery, and operational playbooks rather than memory forensics or traffic failover in general.

1. Classify the failure before retrying

• Recoverable capacity or admission problem: a request arrives when the engine has insufficient KV blocks, active-sequence slots, or memory. If the serving engine supports safe admission refusal, reject/queue within the deadline instead of launching a doomed allocation. A raised PyTorch OOM may be recoverable in some configurations only if the engine explicitly supports the recovery path and its CUDA state remains valid.
• Process-level CUDA failure: illegal memory access, device-side assertion, failed CUDA graph launch, or certain CUDA runtime errors may leave the process's CUDA context unusable. The safest default is to mark that worker unhealthy and replace its process, rather than catch-and-continue without validation.
• GPU/driver hardware fault: NVIDIA Xid, GPU lost/unavailable, repeated ECC errors, or driver reset can require quarantining the affected device or node. Restarting one pod on the same faulty GPU might reproduce the incident.
• Distributed group failure: a single failed tensor-parallel rank, NCCL communicator, or unavailable KV-transfer peer can stall multiple otherwise healthy workers. Recover or replace the whole consistency/parallelism group as required by the serving engine.
• False-positive health failure: a slow model-loading operation, kernel compilation, CPU pause, or network scrape delay can fail an aggressive liveness check. Distinguish genuine unresponsiveness from expected startup work.

Operational rule: If the CUDA context may be corrupted or device health is uncertain, do not resume serving on that worker simply because the next Python exception handler returns.

2. A five-minute triage checklist

1. Identify impact: offered requests versus successful completions, timeouts, 5xx/429/503, stream truncations, p95 TTFT and TPOT, SLO-goodput, and how many compatible replicas remain.
2. Identify the fault domain: one request, model process, pod, GPU, Kubernetes node, tensor-parallel group, or storage/network dependency.
3. Collect logs and metadata: first CUDA error (not just the last propagated exception), model/runtime revision, GPU/driver/CUDA versions, engine settings, prompt-length class, KV-cache use, preemptions and restart events.
4. Check cluster state: pod exit reason/OOMKilled (host RAM), GPU allocatable count, pending pods, NVIDIA device-plugin/GPU Operator logs, and Xid/kernel events.
5. Contain: stop new routing to the affected worker or group; allow safe active streams to drain if the engine can still serve them. Otherwise terminate failed streams with explicit errors and bounded retry behavior.
6. Replace with a known-good model/runtime on a verified healthy GPU; verify readiness by actually exercising the intended inference path.
7. Decide whether it is a local incident or a widespread regression, and whether to roll back the new release or escalate to GPU/driver repair.

Do not log raw private prompts, credentials or retrieved documents during debugging without a specific authorized retention plan.

3. Beginner-friendly numerical example: capacity after a crash

Suppose a 4-replica fleet handles a workload at 6 SLO-compliant requests/s per healthy replica. Production offered demand is 18 requests/s.

• Normal benchmark-safe capacity = \4 × 6 = 24 requests/s\.
• One replica crashes: remaining capacity = \3 × 6 = 18 requests/s\. Theoretical spare headroom is now zero; burstiness and long prompts may violate SLOs even though nominal capacity equals arrival rate.
• A second replica fails: \2 × 6 = 12 requests/s\, leaving a 6 requests/s deficit.
• If demand persists and requests are neither rejected, timed out, nor cancelled, an unbounded waiting queue can grow by roughly \6 × 60 = 360 requests\ in one minute.

An appropriate recovery is not “retry all 360 immediately.” The gateway needs bounded admission, per-request deadlines, fallback policy, and new ready replica capacity. Choose a protected operating limit below benchmark saturation and re-evaluate N−1 resilience.

Memory example: An 80-GiB GPU with 72 GiB already consumed and a new 12-GiB allocation request lacks sufficient nominal free capacity (\80 − 72 = 8 GiB\), even before fragmentation/overheads. Lowering batch/KV demand or scaling out may prevent repeats; blindly increasing memory-utilization flags cannot create physical VRAM.

All numbers are hypothetical teaching examples.

4. Distinguish GPU OOM from host OOM

• CUDA OOM: PyTorch or engine logs show failed device allocation; the process may remain alive, depending on failure handling. GPU memory, KV pressure and CUDA allocations are the relevant diagnostics.
• Linux/Kubernetes OOMKilled: container exits with memory pressure related to host RAM/cgroup limits, potentially including CPU-side checkpoint buffers, tokenization, pinned host memory or KV offload—not automatically GPU HBM exhaustion.
• Startup OOM: loading model weights, CUDA graph capture or workspace allocation exceeds available GPU memory even with zero requests. Check dtype, tensor-parallel size, context/KV reservation, quantization, CUDA graphs and foreign processes.
• Steady-state OOM: active generations, contexts, allocation fragmentation or a real leak push memory over the limit during traffic; follow the quiet-checkpoint and snapshot workflow from Question 269.
• Runtime/kernel fault: an “illegal memory access” may occur with ample free VRAM, so diagnosing it as KV-cache shortage is incorrect.

5. How to debug CUDA errors safely

Use an isolated reproduction, not broad high-overhead debug flags on a customer-facing fleet:

• Start from the first failing CUDA call and its model/request/worker context. CUDA execution is asynchronous, so a later API call might surface an earlier kernel fault.
• vLLM documents \CUDA_LAUNCH_BLOCKING=1\ for localizing the failing CUDA kernel. It changes execution timing and harms performance; use it for controlled debugging only.
• When a crash appears near CUDA Graph replay, vLLM documents \--enforce-eager\ to disable graphs temporarily and better isolate the faulty operation.
• Use NVIDIA Compute Sanitizer or equivalent kernel debugging for reproducible illegal access or race conditions when appropriate; expect substantial overhead.
• For multi-GPU failures, inspect NCCL logs, rank status, topology, driver compatibility and collectives; diagnose whether one rank died before others reported timeouts.
• A version/driver/PTX incompatibility can prevent loading or crash kernels. Compare the supported runtime/CUDA/driver matrix and validate with a small compatible baseline.

Recoverability warning: Some allocation failures can be caught and handled by a serving framework, but illegal memory access and device-side assertions should normally trigger process replacement rather than silently continuing with a potentially invalid context.

6. GPU Xid alerts and device quarantine

NVIDIA Xid codes are generated by the driver to report GPU events. Not every Xid has the same severity, so inspect the code and official guidance:

• Check the affected GPU UUID, timestamp and kernel log (\dmesg\/journald), alongside GPU Operator or device-plugin logs.
• NVIDIA GPU Operator troubleshooting documents a device plugin marking GPUs unhealthy for certain critical Xid errors, reducing Kubernetes allocatable GPU capacity.
• Where configured, GPU health management may mark a GPU lost or tainted and prevent new work from scheduling to it.
• Quarantine suspect devices/nodes on critical errors; replace capacity on other healthy hardware. Follow vendor/cloud procedures for reset, drain, maintenance or node replacement.
• Do not repeatedly clear/restart pods on a device reporting persistent serious Xid/ECC faults. Escalate to SRE/cloud/hardware support.

Kubernetes caveat: mark or drain the correct node/device using your operator's procedures; a generic \kubectl drain\ does not magically migrate running GPU memory or fix a damaged GPU.

7. Recovery workflow for a production deployment

1. Detect and isolate: readiness/health controller withdraws failed engine groups from new traffic. The gateway should not route by container liveness alone.
2. Preserve a healthy pool: protect stable replicas from retry storms using rate limits, exponential backoff with jitter, request deadlines and admission control.
3. Replace safely: schedule a new matching model/adapter/runtime replica on a verified healthy GPU or approved fallback hardware.
4. Warm and validate: wait for weights, graph compilation, KV reservation and a small authorized inference readiness probe. Do not count merely scheduled or process-running replicas as ready serving capacity.
5. Reconnect routing: add the replacement gradually; confirm queue age, TTFT/ITL, completion/error rates, and the fraction of original requests satisfying the SLO.
6. Quarantine bad nodes: hold critical Xid/driver failure devices out of service until cleared with operator/vendor guidance.
7. Run a controlled reproduction: preserve approved crash evidence and determine whether the fault follows requests, software version, GPU device, or workload pattern.
8. Rollback if systemic: if faults follow a new engine/model/hardware configuration, stop its rollout and return to known-good compatible versions.
9. Close with a prevention test: add the initiating failure path to load, cancellation, cold-start, failure-injection or soak suites as appropriate.

A streaming response interrupted by process death generally cannot be resumed at the next token on an arbitrary new worker. Explicitly distinguish client retry/restart from true stateful continuation, and protect tool side effects from duplication.

8. Health probes and crash-loop control

• Startup probe: allow potentially long model checkpoint loading, CUDA graph capture and graph/warmup initialization before applying strict liveness checks.
• Readiness probe: require a working compatible inference engine, not merely an open HTTP port. On an unhealthy event, withdraw *new* traffic quickly.
• Liveness probe: restart genuinely deadlocked/unrecoverable processes; avoid cycling workers that are simply under momentary high load.
• Termination grace: let healthy in-flight streams drain when feasible; no amount of grace fixes a dead CUDA context.
• Restart backoff: avoid repeatedly spawning cold model workers simultaneously; uncontrolled checkpoint downloads can overload storage and inflate recovery time.
• GPU group atomicity: a tensor-parallel group may need coordinated teardown/rebuild if a rank or NCCL communicator is unrecoverable.

Kubernetes and serving-controller health definitions vary; test the actual probes, model loading, routing behavior and failure handling rather than relying on defaults.

9. Production incident: two failures with different fixes

Incident A — expected high-concurrency GPU OOM A chat model starts OOMing when multiple long-context prompts arrive together. Before failures, KV occupancy and active-sequence count rise; no serious GPU Xid errors appear. Restarting clears active allocations, but the issue returns under the same load.

Root cause: Serving concurrency and KV demand exceed a safe memory envelope. Fix: reduce admitted token work or concurrent sequences, select a supported KV allocation configuration, add GPU capacity, and load-test the long-context workload. Verify that cancelled requests free logical KV blocks. Only treat continued memory growth at equivalent quiet checkpoints as a possible leak.

Incident B — GPU fault masquerading as a crash loop After a driver change, one node reports a critical Xid and repeated model-worker failures. Pods rescheduled to the same GPU immediately fail even at light load.

Root cause: Device/driver fault domain rather than heavy inference traffic. Fix: isolate the faulty GPU/node, bring up matching replicas elsewhere, use NVIDIA operator Xid diagnostics, and involve the hardware/cloud team. Do not mask the issue by endlessly increasing pod restart count.

10. Operational metrics and guardrails

Monitor:

• Customer impact: p95/p99 TTFT and TPOT, stream aborts, timeouts, request acceptance and SLO-compliant goodput.
• Worker health: engine uptime, restart count, last fatal error category, ready versus desired replicas, load times and failures during warmup.
• GPU faults: Xid by code/GPU UUID, ECC/device health, allocated versus expected GPU count, driver/runtime version.
• Memory: GPU device free/used HBM, PyTorch allocated/reserved where applicable, KV-cache occupancy, preemptions, host RSS and container OOMKilled events.
• Distributed runtime: failed NCCL collectives, rank exits, prefill/decode/KV-transfer disconnects and recovery time.
• Retries: incoming original requests versus attempted downstream executions; a retry storm can produce high GPU load while success remains low.

Do not:

• Treat a container restart as evidence of successful recovery before verifying healthy inference and SLOs.
• Catch and continue after an unrecoverable CUDA error without a documented safe reset/recovery path.
• Keep scheduling workloads on GPUs marked unhealthy by the device plugin/operator.
• Restart an entire healthy fleet because one replica's KV memory is exhausted.
• Respond to every OOM by increasing GPU-memory-utilization settings.
• Retry streamed tool/action requests blindly; incomplete output and side effects need explicit idempotency/recovery rules.

11. Interview-ready answer

“I would first determine whether the failure is GPU OOM, host OOMKilled, a process-level CUDA fault, an NCCL group failure or a hardware Xid. I'd isolate affected routing, protect healthy replicas with bounded queues and retries, then replace the failed process or group on verified healthy hardware. For OOM I'd examine KV pressure, contexts and allocations; for illegal memory access I'd reproduce with launch blocking/eager mode in isolation; for critical Xids I'd quarantine the device. I'd verify real inference readiness, p95 TTFT/ITL and SLO goodput and add regression/failure tests.”

Research references:

• vLLM — Troubleshooting CUDA and Distributed Serving
• NVIDIA GPU Operator — Troubleshooting Xid and Unhealthy GPUs
• NVIDIA DRA Driver — GPU Health Checking
• PyTorch — CUDA Memory Management and OOM
• Kubernetes — Configure Liveness, Readiness and Startup Probes