How do you diagnose GPU memory leaks, KV-cache fragmentation, and long-running inference worker instability?

Long-running LLM inference workers can become unstable because of genuine memory leaks, legitimate KV-cache growth under load, allocator fragmentation, memory held by CUDA graphs, or problems outside GPU memory entirely. The first task is to classify the failure, not assume that every high-memory reading is a leak.

A true leak means memory remains reachable or otherwise cannot be reclaimed after the workload that required it has finished. Caching intentionally holds reusable allocations; fragmentation leaves free capacity in pieces or pools that cannot satisfy a required allocation; KV-cache pressure means too many active token contexts compete for a finite serving cache. Those situations demand different fixes.

This lesson follows Question 267 (dashboard-level monitoring) and Question 268 (per-request tracing), concentrating on day-2 memory forensics, soak-test reproduction, OOM response, and worker recovery. It does not repeat the purpose of KV caching or the basics of prefix-cache eviction.

1. First distinguish five failure modes

• Expected steady-state allocation: model weights, kernels, CUDA contexts, CUDA-graph private pools, workspace allocations and a reserved KV-cache pool may occupy GPU memory while the service is healthy and idle. High utilization alone is not a leak.
• Dynamic KV-cache pressure: active sequences and their prompt/output lengths consume KV blocks. When workload concurrency or context length rises, the cache fills, new requests wait and the scheduler may preempt/recompute work. Memory should return to the pool as requests finish, although the pool may remain allocated to the worker.
• PyTorch caching allocator: memory reserved by PyTorch may stay above memory allocated to live tensors. The reserved-but-unused portion is not necessarily leaked and can be reused later.
• Fragmentation: there may be sufficient total unoccupied or cached bytes but no suitable allocatable block or usable private-pool capacity for a request. CUDA-graph and eager allocations may be restricted to distinct pools.
• True retention/leak or worker fault: tensors, request states, references, buffers, native allocations, file descriptors, pinned host memory, threads or failing GPU/device/runtime components accumulate after completed requests.

A crashed worker can also result from CPU RAM exhaustion, driver faults, network/device errors, deadlocks or an external termination signal. A restart alone does not prove a GPU memory leak.

2. What to measure at three levels

GPU device / process:

• GPU total/free/used memory, per-process memory, GPU memory errors, temperature/power and worker restarts.
• NVIDIA DCGM or NVML can observe device/process consumption that PyTorch's allocator does not control.

PyTorch/CUDA allocator (where accessible in the worker):

• \torch.cuda.memory_allocated()\: bytes currently held by live tensors tracked by PyTorch.
• \torch.cuda.memory_reserved()\: bytes retained by the PyTorch caching allocator.
• \torch.cuda.memory_stats()\: allocator statistics including active, reserved and inactive split bytes; interpretation depends on the active allocator backend.
• \torch.cuda.memory_summary()\ and targeted memory snapshots: allocation history, OOM events, block splits and allocation stack traces.
• Important visibility limit: PyTorch's memory profiler cannot see *all* CUDA allocations made directly by custom kernels, third-party libraries or other processes. Compare against NVML/device memory.

LLM engine:

• KV-cache blocks in use and available, queue depth, active sequences, cache evictions/preemptions, prompt/output token distributions, request completion and cancellations.
• Engine counters such as \vllm:kv_cache_usage_perc\ and preemption totals are not equivalent to process GPU-memory percentage.
• Model/adapter version, engine configuration, KV dtype, maximum context length and CUDA-graph capture settings.

3. Beginner-friendly numerical example: leak or caching?

Assume an 80 GiB GPU runs a model-serving process. All figures are illustrative and refer to a controlled sample after requests have completed:

• PyTorch live allocated memory: 42 GiB.
• PyTorch reserved memory: 60 GiB.
• NVML-reported process/device usage (simplified): 66 GiB.

Calculate:

1. Allocator-managed *reserved minus live-allocated* = \60 − 42 = 18 GiB\. Some of this may be reusable cached memory; do not immediately call it an 18-GiB leak.
2. Approximate usage outside the reported PyTorch reserved amount = \66 − 60 = 6 GiB\. This may include CUDA contexts, native/third-party allocations and other memory, not necessarily a leak.
3. Nominal device free memory = \80 − 66 = 14 GiB\, assuming no other consumers and ignoring accounting differences.

Now rerun the same fixed workload, allow requests to finish, and compare snapshots at the same quiet point:

• Hour 1: live allocated 42 GiB.
• Hour 2: live allocated 44 GiB.
• Hour 3: live allocated 46 GiB.
• Hour 4: live allocated 48 GiB.

Persistent +2 GiB per hour after equivalent work and cleanup raises suspicion of retained tensors or references. It does not alone prove a leak: verify whether KV usage, CUDA graphs, request length, compilation warmup, adapter caching or allocation pools are still legitimately growing.

If the excess rises at the same rate from 42 to 48 GiB, a naive linear forecast would suggest ~8 further hours to reach 64 GiB allocated. Such extrapolation is only a risk indicator: actual OOM timing depends on other allocations, fragmentation, traffic and the memory budget.

4. Reproduce with a disciplined soak test

1. Freeze conditions: model/weights/tokenizer, runtime build, GPU type, engine flags, quantization, CUDA graphs, scheduler, adapters and routing.
2. Warm up deliberately: let compilation, CUDA graph capture, expected cache growth and dynamic allocations settle before measuring trends.
3. Use a bounded repeated workload: reproduce the same prompt/output length distributions, concurrency, cancellations and streaming behavior across many cycles.
4. Collect time series: device memory, per-process allocated/reserved memory, host RAM, open files, KV occupancy, active and waiting requests, preemptions, p95 TTFT/ITL and failures.
5. Mark quiet checkpoints: wait for active/queued requests to drain; compare memory at *equivalent* workload states, not one idle and one busy.
6. Trigger unusual paths: long-context prompts, client disconnects, timeouts, tool/schema errors, adapter swapping and model revision rollover.
7. Repeat with a known-good version: compare the memory slope and failure count rather than relying on a single run.
8. Capture snapshots near the failure, preferably in a controlled replica. Keep profiler overhead and memory-snapshot retention bounded.

A high watermark that reaches a plateau after warmup is often a healthy cache. A monotonically rising live-allocation baseline across repeated identical completed workloads warrants deeper retention analysis.

5. Diagnose fragmentation with allocator evidence

• Inspect \memory_reserved\, \memory_allocated\, inactive-split stats, allocation size distribution and allocation-history snapshots.
• Check if allocations are trapped in CUDA graph private pools or fragmented into pieces that the requesting allocator cannot reuse.
• Compare OOM logs with actual process/device free memory and allocation sizes. A GPU with free bytes in one pool can fail an allocation requiring another pool.
• Distinguish KV block allocation constraints inside the inference engine from PyTorch's general CUDA heap fragmentation. vLLM's paged KV-cache design mitigates traditional variable-length-contiguous KV waste but does not eliminate block-granularity overhead, capacity pressure or unrelated allocator fragmentation.
• PyTorch \empty_cache()\ releases unused cached blocks from its allocator for other applications; it does not free live tensors, cannot resolve retained references, and is not a blanket production fix. Inference engines may use other memory managers.
• \PYTORCH_ALLOC_CONF=max_split_size_mb:...\ may help some *native allocator* fragmentation situations but carries performance trade-offs. PyTorch documents this as a last resort when inactive split blocks are unusually high; the setting is not meaningful for \backend:cudaMallocAsync\.
• Test allocator changes against equivalent latency, throughput and quality objectives before rollout.

6. Handle KV-cache exhaustion differently

Symptoms:

• KV-cache usage approaches the configured limit during high concurrency or long prompts.
• Waiting requests and preemption/recomputation counts increase.
• TTFT and end-to-end latency deteriorate despite the model weights being loaded correctly.

Engineering actions:

1. Check workload mix, prompt lengths, decode lengths and concurrency; ensure the regression is not just a demand spike.
2. Bound admission, active sequences and token budgets appropriately; for vLLM, settings may include \max_num_seqs\ and \max_num_batched_tokens\.
3. Increase the safe KV-cache allocation only when verified physical GPU memory headroom is available. Raising \gpu_memory_utilization\ without accounting for other allocations can trigger OOM.
4. Scale or reroute eligible long-context requests; maintain latency fairness for smaller prompts.
5. Consider a validated KV dtype/quantization strategy, longer-context allocation design or more suitable parallel topology, with answer-quality and cost checks.
6. Verify that cancellations and finished requests free *their logical KV blocks* promptly. A stale reference to completed sequences is a bug; a large preallocated pool is not.

vLLM explains that insufficient KV cache can cause RECOMPUTE preemption and increased latency; tuning concurrency and cache capacity helps only within the real hardware budget.

7. Practical Python diagnostics

Run this inside a controlled PyTorch GPU worker that actually owns the allocations, not from an unrelated gateway process:

\\\`python import torch

if torch.cuda.is_available(): dev = torch.cuda.current_device() allocated = torch.cuda.memory_allocated(dev) reserved = torch.cuda.memory_reserved(dev) free, total = torch.cuda.mem_get_info(dev) stats = torch.cuda.memory_stats(dev)

gib = 1024 ** 3 print(f"Allocated: {allocated/gib:.2f} GiB") print(f"Reserved: {reserved/gib:.2f} GiB") print(f"Device free: {free/gib:.2f} GiB") print(f"Device total: {total/gib:.2f} GiB") print(f"Inactive split: {stats.get('inactive_split_bytes.all.current', 0)/gib:.2f} GiB") print(torch.cuda.memory_summary(dev, abbreviated=True)) \\\`

Interpretation: \mem_get_info\ is device-wide, whereas \memory_allocated\ and \memory_reserved\ are PyTorch allocator views for the current process. They will not add up cleanly if multiple processes or native CUDA allocations exist. Backend-specific memory stats can be unavailable or zero. A full snapshot or history capture may consume extra RAM and add performance overhead.

For controlled reproductions, PyTorch documents a memory snapshot and visualization workflow: \\\python torch.cuda.memory._record_memory_history(max_entries=100000) # run the bounded test workload torch.cuda.memory._dump_snapshot("gpu_memory_snapshot.pickle") \\\ The leading-underscore snapshot functions are private APIs and may change; consult the installed PyTorch version. Snapshots may contain sensitive allocation stack information and must not be uploaded to public services without approval.

8. Production incident: worker restarts every few hours

Incident: A vLLM deployment successfully serves daytime requests but starts throwing CUDA OOM errors after six hours. Operators see 95% GPU memory usage and assume the KV cache leaked.

Investigation:

1. Compare failed and healthy workers under the same model and trace-replayed workload.
2. After active requests drain, PyTorch live-allocation memory continues rising by ~2 GiB/hour on the affected build; KV-block use returns to its normal baseline.
3. Memory snapshots reveal request-scoped tensors retained in an application-side response-debugging list after cancelled streaming sessions.
4. The GPU also has reserved cache memory, but that stable reserved allocation was not the growing retained allocation.
5. Replaying long streaming requests plus disconnects reproduces the fault; a known-good version does not.

Root cause: A request-lifecycle cleanup bug holds GPU references after cancellations. It is a *true retention issue*, not expected KV-cache residency or fragmentation.

Fix:

• Remove the long-lived references, enforce \try/finally\ cleanup for cancelled streams, release engine-owned request/KV handles, and add a bounded debug record buffer containing safe CPU-only metadata.
• Test cancelled, timed-out and failed requests under soak load; confirm no upward live-allocation trend at equivalent quiet checkpoints.
• Temporarily restore service health by replacing affected replicas through graceful draining, but recognize that periodic restarts only mask the root cause.

Prevention: Add memory-slope and cleanup invariants to worker soak-test gates, alert on unexpected growth *together with* OOM/latency symptoms, and keep release/version identifiers for root-cause correlation.

This is an illustrative incident, not a report of an actual vLLM defect.

9. Recover without creating an outage

• Contain: stop routing new requests to the unstable replica, maintain sufficient ready capacity, and record failed/cancelled streams accurately.
• Drain safely: complete existing work when possible and enforce bounded termination deadlines. Never assume a streaming generation can resume on an arbitrary new worker with no state transfer.
• Restart/replace selectively: use staggered rollouts and warm replacements so a fleet-wide crash loop does not compound model-download pressure.
• Avoid restart storms: cap retries, add backoff, investigate repeated OOM causes, and ensure admission controls protect the remaining pool.
• Use safe circuit breakers: prevent a degraded worker from receiving new long-context workloads while repairs are tested.
• Investigate non-memory causes: host RAM, CPU thread/file-descriptor leaks, device ECC/Xid errors, collective deadlocks, network transport faults, kernel/driver compatibility and worker health probes.

10. Alerts, trade-offs and acceptance criteria

Track:

• GPU device memory, allocator live/reserved/inactive split, KV occupancy and preemption rates.
• Running/waiting sequences, prompt/output length distribution, cancellation cleanup and worker restart/OOM counts.
• Host RSS/pinned memory, open file descriptors, GPU hardware errors, p95/p99 TTFT/ITL and successful SLO goodput.
• The slope of live allocated memory at comparable quiet checkpoints, not just absolute \nvidia-smi\ usage.

Avoid:

• Paging on high but stable reserved GPU memory without failures.
• Calling every allocation failure “fragmentation” without snapshots.
• Increasing KV allocation to the point that weight/graph/workspace memory no longer fits.
• Using \empty_cache()\ as a cure for a real leak or repeatedly restarting the entire fleet during load.
• Treating PyTorch-only snapshots as complete visibility into custom-engine allocations.

Release gate: A candidate should survive representative load, cancellation and soak scenarios with no unexplained retained-memory growth, stable KV return-to-pool behavior, no GPU OOM/crash loops, and acceptable TTFT/ITL/quality. Document the runtime/allocator configuration because results depend on it.

Interview-ready answer: “I would first separate device memory, PyTorch live allocations, reserved allocator memory, and engine KV-block usage. Then I would reproduce the fault with identical load and quiet checkpoints, inspect inactive split blocks and snapshots, and check preemptions and cancellations. A steadily rising live-allocation baseline indicates retention; stable reserved memory may be caching; high KV use with preemptions means capacity pressure. I'd fix the relevant owner, drain unhealthy replicas safely and add soak-test cleanup gates.”

Research references:

• vLLM — Optimization and Tuning: Preemption
• PyTorch — Understanding CUDA Memory Usage
• PyTorch — CUDA Memory Management
• PyTorch — CUDA Memory Statistics
• NVIDIA — CUDA Graph Memory Issues