Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
How do you design SLO-aware autoscaling policies for production LLM inference GPU fleets?
SLO aware autoscaling adjusts the number of ready, compatible GPU serving replicas so real workloads meet latency and reliability objectives at an acceptable cost. Unlike CPU based autoscaling, LLM scaling must account for request/token queueing, TTFT, inter token latency (ITL/TPOT), KV cache memory, prefix reuse, model loading delay, and whole GPU or tensor parallel groups . The controller should add capacity before long queues violate SLOs, remove capacity cautiously, and maintain sufficient headroom for bursts and failures. 1. Why ordinary CPU/GPU utilization autoscaling is insufficient A GPU near 100% utilization can still meet latency targets with effective batching; a GPU at lower utilization might have a long queue or poor cache locality. One 32k token prompt can consume far more prefill work and KV capacity than many 100 token prompts. Request count queue depth is an early signal but not a complete measure of queued tokens or predicted work. TTFT rises with queueing and long prefill; ITL/TPOT rises when decoding becomes congested. They need separate attention. Loading a large model or provisioning GPUs may take minutes; a controller that reacts only after p95 TTFT breaks may be too late. Some model replicas need 2/4/8 GPUs; scale in whole compatible replica groups , not fractional GPUs. 2. Four metrics and the control policy Leading indicators: offered incoming prompt/output token work rate, waiting prompt tokens, active sequences, queue age, KV block utilization and available ready replicas. Lagging or validation indicators: observed p95/p99 TTFT, ITL/TPOT, joint SLO compliant goodput, deadline misses and rejection/error rates. A useful policy combines: 1. Capacity planning baseline: benchmark goodput per replica for the actual workload mix. 2. Queue/KV pressure triggers: add replicas when sustained queued work or KV occupancy predicts imminent TTFT/ITL violations. 3. SLO guardrails: increase capacity or enable overload protection when p95 latency or deadline errors cross thresholds. Measure under enough request volume to make percentiles meaningful. 4. Scale down safety: keep a lower threshold and a stabilization/cooldown window to avoid oscillation, then drain idle capable replicas without interrupting active streams. KServe documents Prometheus/KEDA autoscaling on vLLM waiting request and KV cache metrics; vLLM Production Stack offers a KEDA/Prometheus integration. NVIDIA Dynamo Planner describes SLO driven scaling with latency and KV telemetry. Treat these as distinct supported implementation options, not universally equivalent features. 3. Beginner friendly numerical example Suppose one ready GPU model replica sustains 6 successful SLO conforming requests/second for the measured production workload at the chosen target operating level, with TTFT ≤ 1 second and TPOT ≤ 50 ms/token . Normal incoming load = 12 requests/s , so ceil(12/6)=2 replicas are required. An upcoming traffic burst is forecast to reach 25 requests/s . Minimum needed at the same safe capacity = ceil(25/6)=5 replicas. Add 3 ready replicas before the burst if model warmup and GPU provisioning time are significant. If one replica failure tolerance must also be preserved during the burst, provision 6 replicas , because losing one still leaves 5 × 6 = 30 requests/s safe modeled capacity. At an illustrative $2/GPU hour and one GPU per replica, cost rises from $4/h for 2 replicas to $10/h for 5, or $12/h for 6 with failure headroom. These prices are hypothetical, not live cloud rates. Critical nuance: ceil(load / per replica goodput) is a planning approximation, not a guarantee. Traffic variance, queueing, token mix, cache warmness, uneven routing and actual acceptance/error policies determine achieved SLOs. 4. A concrete hybrid scaling rule Start with a measured safe per replica capacity c, observed/forecast offered work rate λ, minimum replicas r min and configured headroom: r demand = ceil(λ / c). r queue = ceil(total queued token work / desired queue work per replica) or another calibrated backlog estimator. r target = clamp(max(r demand, r queue, r kv, r slo), r min, r max), where extra terms reflect calibrated KV and SLO pressure. Apply fast, rate limited scale up , slower scale down with hysteresis , and a configurable maximum number of GPUs provisioned per decision. This is a conceptual controller design—not a direct KEDA or Kubernetes configuration. Do not add uncalibrated metrics with incompatible units. A queue cap suitable for 512 token messages may be ineffective for 32k token document prompts. Scale down example: if traffic falls from 25 to 12 RPS, do not immediately remove 3 replicas. Keep the desired count until an adequate low load window passes, then stop routing new requests to selected replicas, complete/cancel existing work according to deadlines, release KV state and terminate them safely. 5. Startup, model warmup and bursts Track GPU node provisioning + image pull + model weight loading + readiness + warmup as one time to ready distribution. Predictive prewarming helps known daily/weekly surges; reactive scaling may be too slow for a sudden 30 second burst. A replica should receive new requests only after model/version/tokenizer/adapter compatibility, KV allocation and readiness are confirmed. While new capacity is unavailable, use bounded queues, admission control and appropriate 429/503 responses; unbounded queueing does not preserve the SLO. Reserve some ready capacity or use pool level scheduling for predictable SLOs; trade increased idle GPU cost against user latency and reliability. 6. Multi GPU replicas and disaggregated models A tensor parallel serving group using four GPUs is one replica for admission purposes , not four independent request serving replicas. A disaggregated deployment has distinct prefill , decode , KV transfer and sometimes router bottlenecks. Tune prefill and decode counts using their own bottleneck telemetry; scale one role without overproducing KV transfers for a saturated downstream role. Some infrastructure controllers scale complete tensor parallel groups as atomic units. Protect the network and model store limits that might delay simultaneous launches. Question 244 covers detailed sizing of individual disaggregated prefill/decode pools; this lesson focuses on fleet wide SLO based scaling controls, warmup, hysteresis and economics . 7. Production incident: autoscaler reacts too late Incident: A chat service runs two one GPU replicas, both at high decode occupancy. A burst increases demand from 12 to 25 RPS . The controller scales only when aggregate GPU utilization stays above 90% for several minutes. New model pods take 3 minutes to become ready; meanwhile p95 TTFT climbs from 700 ms to 5 seconds , queued requests grow and retries amplify overload. Root cause: GPU utilization was a lagging/insufficient signal, queue tokens were unmeasured, and the controller did not consider model startup lag. Fix: Establish safe per replica goodput, alert/scale on early queue token/KV pressure, prewarm replicas for forecast peaks, use fast scale up but bounded GPU provisioning, propagate cancellation and enforce token aware admission during the warmup gap. Prevention: Replay realistic bursts and replica failures; verify both the time to ready and the recovered p95 TTFT/ITL. Test downscale/draining so live generations are not interrupted. 8. Testing and operational guardrails Run closed loop and open loop tests, emphasizing offered RPS rather than only completions. Record p50/p95/p99 TTFT, ITL/TPOT, joint SLO compliant goodput, deadline misses, 429/503, pending token work and KV occupancy. Measure autoscaler recommendation vs actual ready replica count, pending GPU nodes, cold start time and resource quotas. Compare time to scale, queue overshoot, stability/flapping, cost per SLO successful request and N−1 recovery. Break down metrics by tenant/model/context length so one busy fleet does not conceal another fleet's failures. Apply rollout safeguards and version checks: a KEDA trigger expecting particular vLLM metric names may not work after incompatible engine/adapter upgrades. Rollback rule: if p95/99 SLO breaches, queue growth or cost escalation exceeds the preset threshold, revert to a safe minimum pool size/policy and inspect metric accuracy. 9. Interview ready answer “I would benchmark safe SLO goodput per compatible GPU replica, forecast or observe offered token work, then scale on early queued token and KV pressure with TTFT/ITL guardrails. I'd account for model startup time, keep N−1 headroom where required, scale whole multi GPU replicas, and downscale slowly only after draining. I'd evaluate p95 TTFT, TPOT, goodput, errors, time to ready and cost per good request, rather than CPU/GPU utilization alone.” References: KServe — Autoscale InferenceService with LLM metrics vLLM Production Stack — KEDA Autoscaling NVIDIA Dynamo — GPU inference autoscaling Google Cloud — Best practices for LLM inference autoscaling
How do GPU cold starts, model loading, and prewarming affect LLM inference autoscaling?
GPU cold starts are the delays between deciding to add an LLM inference replica and having that replica ready to serve compliant requests . They can dominate autoscaling response time because the system must obtain GPU capacity, fetch or mount checkpoint shards, load weights into accelerator memory, initialize the runtime, compile/capture kernels, and confirm serving readiness. Prewarming moves some or all of this work ahead of the burst, trading extra memory and idle GPU cost for lower time to ready and fewer missed latency SLOs. Question 261 designed the overall autoscaling policy. This lesson narrows in on startup path bottlenecks, model loading optimizations, readiness, prewarming, and the economics of reserve capacity , rather than restating general scaling triggers. 1. Break down the end to end cold start path 1. Detect demand: monitoring/forecasting interval plus autoscaler decision delay. 2. Acquire capacity: Kubernetes scheduling, GPU node provisioning or allocation, driver/runtime readiness, and placement of the correct multi GPU topology. 3. Prepare the container: image pull, startup, secrets/config and network/storage mounts. 4. Stage model weights: download from object storage or read from local/shared filesystem; fetch tokenizer and model/adapter metadata. 5. Initialize the model: deserialize/checkpoint map shards, allocate and load parameters into GPU HBM, configure tensor parallelism/quantization and reserve KV/cache/workspaces. 6. Optimize engine startup: kernel compilation, torch.compile, CUDA graph capture and any runtime specific preparation. 7. Warm up and become ready: exercise supported shapes, verify a simple inference, advertise version/adapter compatibility, then allow the router to send traffic. These phases sometimes overlap. Critical path duration is not necessarily the sum of all observed stage timers when work executes concurrently. 2. Beginner friendly numerical example A hypothetical service experiences the following serial cold start times for one two GPU model replica : GPU placement and node readiness: 80 s Container and configuration: 20 s Checkpoint transfer/load: 70 s Engine compilation and capture: 40 s Warmup/readiness: 10 s Total = 80 + 20 + 70 + 40 + 10 = 220 seconds , approximately 3 minutes 40 seconds before the new replica can serve. Now suppose a burst arrives in 30 seconds : Reactive scale out cannot make this cold replica ready by the burst. If the existing fleet has 12 SLO safe requests/s and burst demand reaches 24 requests/s , the instantaneous shortfall is 12 requests/s until capacity becomes available. Ignoring cancellations, early rejection and transient demand changes, a completely unbounded queue could add approximately 12 × (220 − 30) = 2,280 requests across the remaining 190 seconds if demand stayed constant and the new replica resolved the deficit upon readiness. A realistic production system should bound queued work and shed/defer requests , not accumulate thousands beyond their deadlines. 3. Compare warm start strategies A. Keep a fully ready spare A GPU replica already loaded, warm and in the router's compatible capacity pool can accept eligible work almost immediately. Best latency protection , but incurs ongoing reserved GPU/HBM cost and potential wasted idle capacity. B. Keep a preloaded checkpoint on node local NVMe/shared PVC Avoid repeated remote downloads and registry/object storage bottlenecks. Still pay GPU scheduling, model construction, disk to HBM transfer, compilation and readiness costs. C. Cache compilation/engine artifacts Reuse compatible compiled kernels or model metadata to reduce repeated setup. Cache validity depends on GPU architecture, model settings, parallel topology, dtype and runtime version. A warm start with cached artifacts is different from an already running, fully warm replica. D. Weight preloading and persistent GPU residency vLLM's preload feature uses a daemon holding per GPU weights and an engine ipc cache loader, subject to fingerprint and GPU UUID compatibility checks; a cache mismatch falls back to normal disk loading unless configured otherwise. This is an implementation specific optimization and does not mean every vLLM deployment or GPU tenancy model can use it . E. Preinitialized shadow engine / standby NVIDIA Dynamo documents shadow engine recovery as a specialized, preview failover design that keeps initialized standby state near the active engine. A reported B200 case recovered in 7.3 s versus 283 s for a cold restart. This is engine recovery after failure , not necessarily provisioning an entirely new GPU replica for demand scaling. Do not equate its failover speed with general autoscale startup time. 4. Memory and bandwidth fundamentals For a checkpoint containing 70 billion parameters at 2 bytes/parameter , weight storage alone is approximately: 70,000,000,000 × 2 = 140,000,000,000 bytes ≈ 130.4 GiB. At an ideal aggregate 5 GiB/s effective transfer rate , moving 130.4 GiB takes at least 26.1 seconds . Real initialization can be much longer due to per file I/O, decompression, CPU construction, data layout, network contention, checkpoint sharding and GPU allocations. Parallel tensor sharded loading may spread bytes and work across several GPUs but adds communication/coordination complexity. Beware shared storage stampedes: if 10 replicas simultaneously read a 130.4 GiB checkpoint from the same storage tier, the fleet may demand about 1.3 TiB of reads if no effective cache or deduplication exists. This can saturate network/storage and cause scale out to become slower as more replicas are requested . Stage weights, limit concurrent launches or use a distributed model cache. 5. Prewarm based on demand and startup lead time 1. Measure p50/p95 time to ready by model, GPU group, node pool and cache state. 2. Forecast token work demand and spike timing; issue scale up sufficiently before peak when demand is predictable. 3. Maintain a minimum warm replica pool for high priority, latency sensitive workloads. 4. Keep startup in progress visible separately from ready capacity ; Kubernetes desired/created replicas are not ready serving replicas. 5. Use safe scale up waves when model storage or GPU provisioning is rate limited. 6. Scale down cautiously after sustained low load, drain existing streams and release KV state; remove the correct number of GPU groups rather than killing workers mid generation. 7. Review idle warm pool cost against reduced SLO violations and avoided rejected requests. Example: if measured p95 end to end startup is 240 s , a forecast peak at 12:00 suggests starting scale up no later than roughly 11:56 , plus a small operational buffer. For an unpredictable 15 second spike, only already ready capacity or bounded overload behavior can protect immediate traffic; cold GPU provisioning is too slow. 6. Distinguish readiness from liveness Liveness: the process/container is running; not evidence that the correct model weights are loaded. Readiness: model/version/tokenizer/adapter compatible, engine initialized, GPU/KV memory allocated, and supported inference path usable. Avoid routing production traffic merely because the pod is running or the TCP port is open. Probe readiness after full model initialization; ideally verify a small inference under a controlled check without creating excessive load. Prevent premature autoscaler scale down of newly launched but not yet ready workers due to misleading utilization or idle metrics. 7. Production incident: scaling creates a model download storm Incident: A chat service has two ready GPU replicas and forecasts a high load period poorly. At noon, demand doubles. Autoscaler requests six additional replicas, all downloading the same very large checkpoint from one shared storage backend. Network throughput collapses, startup jumps from 3 minutes to 9 minutes , queues grow and user p95 TTFT exceeds its target. Debug: Trace autoscaler recommendation → pod scheduled → image ready → checkpoint available → HBM loaded → engine compiled → first successful inference. Inspect object storage request latency, disk throughput, shared PVC IOPS, GPU allocatable capacity, weight download retries, compilation cache hits and readiness probe transitions. Compare time to ready for one instance versus many launching concurrently. Root cause: The policy scaled on desired replica count but did not account for weight distribution contention or distinguish cold/ready capacity. Excessive parallel starts caused a storage stampede. Fix: Stage checkpoint shards in a shared or node local cache, use compatible parallel loading configuration, and control startup concurrency. Prewarm enough fully ready replicas ahead of expected peaks. Drive admission decisions from ready GPU capacity, not replicas requested. Maintain bounded queues/deadlines and cancel abandoned work while new capacity becomes available. Prevention: Benchmark cold versus warm startup with multiple simultaneous replicas, test node replacement and cache misses, track p95 readiness latency, and enforce storage/GPU quota headroom. 8. What to measure and how to choose a strategy Cold and warm startup p50/p95/p99 , total decision to ready time and breakdown per stage. Weight transfer throughput (GB/s), storage IOPS, cache hit rate and concurrent model load saturation. Warm capacity cost: paid GPU hours, standby memory and useful SLO compliant requests. SLO outcomes: tail TTFT/ITL during bursts, admission rejections, queue token depth and time to recover. Reliability: image/model version mismatches, failed GPU allocations, compilation errors, readiness probe failures and failed rollbacks. Economics: balance cost of maintaining spare ready GPUs against the measured probability and cost of cold start SLO violations. 9. Edge cases and limitations A scale to zero design may be economical for asynchronous or noninteractive traffic but is usually incompatible with strict sub second response objectives unless an appropriate warm layer or fallback exists. Quantized or mixture of experts model loading may involve specialized metadata, packing, expert placement or kernel preparation; raw checkpoint size does not capture all latency. Some model revisions invalidate compilation caches or weight fingerprints. GPUs on the wrong node topology or in another region cannot always be assembled into the required tensor parallel replica. Persistent warm weights/standby engines consume memory or limit GPU sharing; validate isolation/security requirements. Avoid blindly reusing shared KV or model caches across incompatible adapters/models or tenant security boundaries. Interview ready answer: “I'd measure every stage from autoscaling decision to an inference ready model, not merely pod creation. Checkpoint download, HBM load and compilation can take minutes, and starting many replicas together can saturate shared storage. I'd maintain a warm minimum pool, preload/copy weights where supported, cache compiled artifacts, prewarm based on p95 startup lead time, and gate routing on readiness. I'd compare SLO violations during bursts against the GPU hour cost of spare capacity.” Research references: IBM Research / MLSys 2026 — Breaking the Ice: Analyzing Cold Start Latency in vLLM vLLM — Preload vLLM — Cold/Warm Startup Benchmark NVIDIA Dynamo — Model Caching NVIDIA Dynamo — Fast Model Weight Loading NVIDIA — Shadow Engine Recovery in Dynamo (August 2026)
How do you safely drain, replace, and roll out LLM inference replicas without interrupting streaming requests?
Graceful draining lets existing LLM generations finish while preventing new requests from entering a replica scheduled for replacement. A safe rollout couples routing exclusion , bounded in flight completion , sufficient replacement capacity , and observable rollback . Inference requests are special because a streamed response may stay open for many seconds or minutes and hold scarce GPU/KV cache state. Merely removing a pod from a load balancer does not guarantee its existing streams survive. This is different from Question 262 (making cold replicas ready) and Question 253 (cancelling unwanted work). Here the objective is to preserve useful active work during deliberate rolling upgrades or scale down . 1. Why standard rolling updates can interrupt LLM streams Long prompts and lengthy decode sequences create request lifetimes longer than ordinary HTTP transactions. Once an SSE or gRPC streaming connection is established, it is typically attached to a specific engine and KV cache state. Killing the old replica during a token stream can produce a truncated answer, a client visible disconnect and discarded KV blocks. A new replica cannot generally reconstruct and continue the same generation seamlessly unless explicit state migration/resume is supported and validated. New replicas may have different model weights, tokenizers, adapters, cache layouts or safety policies; blindly replaying a partial response can duplicate tokens or tool side effects. 2. Safe deployment sequence 1. Provision replacement first: start enough new replicas, load model checkpoints and adapters, warm up the attention/runtime paths, then confirm true inference readiness—not just container liveness. 2. Check version compatibility: route requests only to replicas with the correct model revision, context window, LoRA adapters, region and security requirements. 3. Mark the old replica draining: make it ineligible for new sessions at the gateway/router. Validate that Endpoint Picker, load balancer and all ingress paths have observed that state. 4. Keep existing streams alive: allow running requests to decode on the old engine, retaining their KV blocks and connection paths. 5. Wait for an in flight counter or deadline: track active/queued requests and outstanding SSE streams; avoid creating new queues on the draining replica. 6. Apply a termination ceiling: after a configured grace period, explicitly cancel/close the remaining requests with clear error semantics rather than allowing uncontrolled process termination; longer grace protects long jobs but consumes billed GPU time. 7. Terminate only after drain completion or timeout , then release GPU/KV allocations, unload checkpoint and record success/failure metrics. 8. Roll out gradually: monitor the canary and proceed from a small traffic fraction to larger fractions only when quality, p95/p99 TTFT and streaming completion are healthy. 3. Beginner friendly numerical example: choosing a drain budget Assume an old inference replica has 10 active streaming requests when the operator initiates scale down: 6 requests are estimated to complete within 15 seconds . 3 requests need 40 seconds . 1 unusually long request needs 120 seconds . Suppose the application uses a 60 second Kubernetes termination grace period , with 15 seconds reserved for traffic propagation/preStop handling and 45 seconds for engine draining. Illustrative result: The six 15 second requests and three 40 second requests may finish normally if their estimated durations hold and the termination signal/transport are handled correctly. The remaining 120 second generation will not finish in 45 seconds and needs an explicit cancellation/error path , a longer grace policy or a workload specific route to a non draining pool. 9 of 10 active requests finish within the 45 second drain window in this simplified example ( 90% ), but the result is not guaranteed by the timeout setting alone. Setting a 120 second grace may allow the last request to finish, but raises the period during which the draining GPU stays unavailable for new traffic and still incurs cost. KServe v0.19 example: its release documentation describes a preStop sleep of 15 seconds to let routing detect endpoint removal, and a vLLM shutdown timeout derived from the remaining grace period (45 seconds for a 60 second example). These are implementation specific settings , not Kubernetes wide defaults. <https://kserve.github.io/website/blog/kserve 0.19 release 4. Why preStop and grace period accounting matter Kubernetes invokes a preStop hook before TERM , and that hook's runtime counts against the pod's overall termination grace period . With terminationGracePeriodSeconds: 60 and a 15 second preStop , the application has roughly 45 seconds left—not another full 60 seconds—before forced termination. Terminating EndpointSlice endpoints have ready=false; a separate serving condition may indicate whether existing connections can still be served. A router/load balancer must actively respect the terminating/draining state and its own connection drain behavior; do not rely on readiness toggles alone for every external gateway. SIGTERM handlers should stop new admission, finish or cancel pending work, and exit within the remaining budget. A hanging preStop or overly short grace triggers SIGKILL. Do not confuse liveness and readiness: removing a drained worker from routing is a controlled action, not a crash requiring immediate restart. 5. Rolling update capacity and availability calculations Assume a deployment with 4 replicas that can each deliver 5 SLO compliant requests/s on the current traffic mix: Total benchmark safe serving capacity = 20 requests/s . Live demand = 16 requests/s . If one replica is removed before a replacement becomes ready, capacity falls to 3 × 5 = 15 requests/s , below demand . The queue grows and new users may miss TTFT SLOs. A surge replica made ready before terminating the old worker preserves at least 20 requests/s during the transition, assuming routing is effective. If long lived streams keep the old replica busy, count its GPU usage but do not assume it is available to accept new traffic . Capacity accounting must distinguish active/draining and ready for new requests. Capacity planning should consider Kubernetes maxSurge / maxUnavailable, PodDisruptionBudget or equivalent protections, GPU quota and topology availability. These settings are not a substitute for actual inference readiness and SLO checks. 6. Canary and progressive rollout Start new model/server versions alongside stable replicas; validate startup, health and one real inference path before routing substantial traffic. Send a small, explicit traffic fraction to the candidate while keeping stable capacity ready for rollback. Observe stream truncation/disconnect rate , p95/p99 TTFT/ITL, queue delays, KV pressure, error rate, structured output quality and tool/schema correctness. Increase the candidate weight gradually only if it passes the predeclared gates. If quality or serving health regresses, stop new requests to the candidate and drain its existing streams when safe; route new requests back to the stable group. KServe's canary rollout documentation describes weighted routing groups for LLMInferenceService, with weights representing relative share , not a literal percentage field. Verify the feature in the deployed KServe version. A canary is not a proof that old open streams can migrate. Treat each established streaming connection as bound to its serving engine unless an explicit resume feature exists. 7. Production incident: rollout cuts off answers Incident: During a vLLM image update, 20% of long running streamed answers end early with network disconnects. A new pod becomes ready, but the old pod is killed after a short default shutdown period. Root cause debugging: 1. Correlate rollout timestamps with streaming disconnects and request completion codes. 2. Inspect pod events, preStop duration, SIGTERM/SIGKILL timing and actual terminationGracePeriodSeconds. 3. Check router logs: did it stop assigning new requests to terminating pods? 4. Check engine active requests, decode durations, KV occupancy and the distribution of stream lifetimes. 5. Confirm the new replica had fully loaded the right model/adapter and sufficient SLO capacity. Fix: make drain state visible to the router; allow routing propagation before TERM; use a shutdown timeout sized to real active stream durations; ensure replacement capacity is ready; preserve model/version compatibility; introduce canary traffic and rollback thresholds. Prevention: regularly test deployment and scale down during long SSE/gRPC streams. Run forced termination chaos tests to confirm graceful errors and no duplicate tool side effects. 8. Trade offs and operational guardrails Longer drain grace improves completion of lengthy requests but delays freeing GPUs and can slow rollouts. Shorter grace frees capacity quickly but increases stream termination and client retries. Drain versus retry: retrying a partially streamed response may produce different text. A generic client retry is not a continuation protocol. Stateless ingress versus stateful KV cache: frontend load balancing cannot usually transfer the active decoder's KV memory to another arbitrary worker. Disaggregated prefill/decode: drain coordinated worker groups or guarantee that existing decode workers still have the necessary transferred KV state; don't tear down a shared dependency prematurely. Storage and model startup contention: starting several surge replicas at once can create a model download storm; coordinate with prewarming from Question 262. Security: maintain authenticated model routing and avoid crossing tenant/region boundaries during rollback or failover. 9. What to monitor Draining replicas, active sessions at drain start/end, stream completion and forced cancellation counts . Time from draining announcement to last new request arriving on the old replica. Time to drain p50/p95/p99 by output length bucket, plus grace period expirations. Unexpected SSE/gRPC disconnects, partial output, duplicate tool effects and client retries. Ready for new traffic capacity, GPU quota, queue tokens, KV occupancy and p95 TTFT/ITL. Canary versus stable failure, latency and quality metrics; rollback frequency. Interview ready answer: “I'd bring up and warm a compatible replacement before draining the old LLM replica. Then I'd remove the old endpoint from new request routing, keep its current streaming generations alive, and give the engine a bounded drain period covered by Kubernetes termination grace. I'd watch active requests, stream truncation rate, p95 latency and ready capacity while rolling out gradually. A live stream cannot normally be moved to a different replica just by changing the load balancer, so expiry needs an explicit cancellation/recovery policy.” Research references: KServe v0.19: graceful shutdown with vLLM drain Kubernetes: Pods and Endpoint termination flow Kubernetes: container lifecycle hooks KServe LLMInferenceService canary rollout Kubernetes: Pod lifecycle
How do you perform canary releases and automated rollbacks for LLM inference models using latency, quality, and reliability signals?
Canary releases gradually expose a small, controlled portion of production LLM inference traffic to a new model, serving engine, or configuration while retaining an existing stable deployment. Automated rollback stops sending new requests to a candidate when predefined latency, answer quality, or reliability gates fail. Because LLM changes can harm correctness even when HTTP status codes remain 200, a robust canary needs both serving telemetry and task level quality evidence. This lesson builds on Question 263 (preserving active streams during replica replacement) but focuses on experiment design, promotion criteria, automated decision rules, statistical confidence, rollback and quality rather than shutdown mechanics. 1. What might change in an LLM release? Model or tokenizer revision: may change answer quality, token lengths, formatting, context window, tool calls and latency. Inference engine: vLLM version, attention kernels, continuous batching, scheduler, tensor parallelism or cache policy can change TTFT/ITL and reliability. Precision: weight/KV quantization or speculative decoding can change performance and output distribution. Prompt and retrieval configuration: prompt template, system policy, RAG tool settings or adapter changes can affect quality even with unchanged weights. Routing and GPU topology: hardware or load balancing changes can alter request queues, throughput and failure domains. Some serving platforms allow version traffic splitting only between backends with the same external model name and adapter set . In particular, the documented KServe LLMInferenceService weighted group has these compatibility requirements; use an appropriate higher level gateway or a different deployment strategy when comparing genuinely different named models. 2. A phased production rollout 1. Offline gate: run reproducible datasets for correctness, safety, JSON/schema validity, task completion and long context behavior. Reject regressions before user exposure. 2. Shadow phase where allowed: duplicate approved requests to the new candidate without returning shadow outputs to the user . Account for privacy, extra GPU cost and side effecting tools (disable or sandbox external actions). 3. Prewarm candidate: verify model/adapter identity, inference readiness, GPU/KV headroom, and version specific metrics. 4. Canary small: send a limited share of eligible traffic (for example 5% ); ensure the canary is not overwhelmed because its GPU pool is underprovisioned relative to its weight. 5. Compare equivalent traffic: measure per version p95/p99 TTFT, TPOT/ITL, error rate, SLO compliant goodput, timeout/rejection, KV occupancy, and workload specific quality. 6. Promote deliberately: increase traffic in stages (illustratively 5% → 10% → 25% → 50% → 100% ) after minimum exposure and predefined gates. 7. Abort or roll back: promptly stop new traffic to an unhealthy canary; keep stable capacity ready and drain or cancel ongoing candidate streams according to the verified shutdown policy. Traffic weights are relative , not inherently percentages: KServe's LLMInferenceService documents 9:1 = 90%:10%, and 9:9 = 50%:50% when both members are eligible. Its version support and routing prerequisites must be checked. 3. Beginner friendly numerical example Suppose a new vLLM engine version receives 10% of 10,000 eligible requests during an analysis interval: about 1,000 canary and 9,000 stable , ignoring routing variance. The published rollout rules require: p95 TTFT ≤ 1.2 seconds HTTP error rate ≤ 1% structured output validity ≥ 99% task specific accuracy loss ≤ 1 percentage point , on a sufficiently powered, independently evaluated holdout. Illustrative observed metrics: Stable: p95 TTFT 0.85 s ; errors 45/9,000 = 0.5% ; valid JSON 99.6% . Canary: p95 TTFT 1.50 s ; errors 20/1,000 = 2.0% ; valid JSON 98.2% . Decision: the canary fails three predetermined gates (TTFT, errors, JSON validity), so halt and roll back . The 2% versus 0.5% error rate is a 1.5 percentage point absolute increase , not merely a 1.5% relative change. Because the canary samples fewer requests than stable, estimates have sampling uncertainty; don't automatically promote a candidate after an inconclusive low traffic sample. 4. Quality is more difficult than latency Immediate online checks: invalid JSON/schema, truncation, tool call contract violations, safety policy violations, refusal anomalies and a small set of deterministic correctness checks. Delayed labels: user ratings, task completion, human audits or real world downstream outcomes may arrive minutes to days later. Paired shadow evaluation: compare candidate and stable answers to the same prompts , when legally and operationally permitted, reducing traffic mix confounding. LLM as judge scores need rubrics, bias checks and human calibration. Safety gates: critical privacy, access control or unsafe tool use regressions may require immediate rollback, even with a tiny sample. Do not treat output differences as automatic defects: stochastic decoding can produce different yet valid answers. Prefer task success and correctness evaluations. If the model itself changes, decide whether the rollout is testing inference performance at comparable quality or intentionally changing model quality. Apply the right launch approval and user facing fallback contracts. 5. Automated decision gates without noisy false positives Use a predeclared decision matrix: Hard abort: security/tenant data leak, unauthorized side effect, incompatible tokenizer/tool schema, model loading failure or severe correctness regression. Latency/reliability: p95/p99 TTFT or ITL breaches, error rate ratio/absolute rate limits, queue spikes, streaming disconnects and SLO violations sustained for an evaluation window. Quality: sufficient evidence of reduced task accuracy, structured output validity or policy adherence. Inconclusive: insufficient sample size, missing metrics, an unequal workload mix or the candidate not yet warmed. Pause , extend observation or gather more data; don't automatically claim success. Promotion: all required metrics pass over the declared window with sufficient exposure and no safety red flags. Use guardrail thresholds calibrated to real baseline variance. Percentile estimates with a few dozen canary requests can be unstable. Compare by model, prompt length, tenant, region and traffic priority, while limiting simultaneous changes. Argo Rollouts has AnalysisTemplates and AnalysisRuns for Prometheus metric checks; analysis outcomes can permit progression, abort on failure or pause on an inconclusive result. KServe documents per version Prometheus labels and queries, necessary to compare the candidate with stable accurately. A weighted traffic split alone does not provide automated quality gated rollback without a suitable analysis/controller layer. 6. How to observe each version reliably Monitor: TTFT: histogram quantile(0.95, sum by (le, llm isvc name) (rate(vllm:time to first token seconds bucket[5m]))) Error rate: failures divided by attempts, scoped to version, with expected cancellations and rejection policies documented. Streaming: p95/p99 inter token latency, truncated streams, end to end completion and token counts. SLO compliant goodput: successful requests meeting every configured latency target per second; compare with acceptance and errors to avoid gaming the metric. Queue and KV: waiting requests/tokens, active sequences, cache occupancy, preemption and GPU load. Quality: invalid structured output, tool call failures, offline accuracy score and shadow/human evaluation. Metric label caution: the llm isvc name tag in KServe's guide is added through PodMonitor relabeling ; it is not always automatically present. Check actual labels before writing promotion rules. Percentile aggregation must combine histogram buckets before calling histogram quantile, not average already computed per pod p95 values. 7. A production incident Incident: A new inference runtime is deployed to 20% of traffic because it improves raw output tokens/s by 18%. Within an hour, p95 TTFT rises from 700 ms to 1.9 seconds for long prompts and tool call JSON validity falls from 99.7% to 96% , although the HTTP error rate remains low. Root cause: rollout checked process readiness and 5xx responses but not long context latency or task level output validity . The canary's model/engine optimization changed scheduling and structured output behavior. Fix: 1. Stop new canary routing; retain healthy stable capacity. 2. Drain existing canary streams rather than killing active generation indiscriminately. 3. Re run long context and tool schema regression suites; examine scheduler settings and runtime changes. 4. Re release only after correcting issues and adding per length quality plus TTFT gates. Prevention: Build automated multi signal analysis, keep rollback paths warm, predefine sample sufficiency and test the rollback itself under live streaming. 8. Operational trade offs Exposure versus detection: a tiny canary reduces blast radius but yields fewer samples; long observation improves confidence but prolongs exposure. Cost: running old and new replicas concurrently uses extra GPUs; saving standby cost too soon destroys fast rollback. Route consistency: tools/session state and prefix/KV caches may be version specific; pin long lived sessions where required and never assume decoder KV can migrate between models. Traffic mix: a 10% slice is not statistically equivalent if all long prompts or premium tenants land on one arm. Fallback safety: an engine downgrade is not guaranteed to preserve quality, output tokens or tool semantics. Security, privacy and data residency still apply. Analysis data gaps: automated rules must explicitly handle missing telemetry, low counts and service failures so that a metrics outage does not silently approve a risky release. 9. Interview ready answer “I would deploy a warmed candidate alongside stable replicas, gate it on offline correctness, and shift a small amount of compatible traffic while collecting per version latency, streaming, error and quality measures. The rollout should advance only after adequate samples satisfy predefined p95 TTFT/ITL, SLO goodput, JSON/tool validity and safety criteria; a hard correctness or policy failure immediately aborts. An analysis controller can automate rollback, while the router stops new candidate traffic and drains existing streams. I'd validate the entire rollback path before the release.” References: KServe — Canary Rollout for LLMInferenceService KServe — Observability for Canary Rollouts Argo Rollouts — Analysis and Progressive Delivery Argo Rollouts — Architecture
How do you implement shadow traffic and offline-to-online validation for new LLM inference deployments?
Shadow traffic (traffic mirroring) sends a controlled copy of an eligible production request to a candidate LLM deployment while the stable deployment continues serving the real user. The candidate's result is never delivered to the user during shadow evaluation. It lets engineers test real prompt lengths, traffic bursts, latency, output quality and backend stability before an actual canary release. It is not a substitute for canary testing: users do not see candidate answers, and shadow traffic cannot reveal every feedback, session or downstream workflow effect. Question 264 covered progressive live canaries and automated rollback. This lesson covers the earlier offline evaluation → shadow replay → paired analysis → limited canary validation sequence. 1. The complete validation ladder 1. Offline functional checks: unit/contract tests for tokenizer, chat template, API schema, prompt compatibility, model and adapter loading, context limits, JSON/tool output and safety constraints. 2. Offline quality evaluation: stable versus candidate on an unchanged, versioned holdout with task specific accuracy, reasoning/code tests, long context retrieval and privacy/security cases. Repeated generation may be needed for stochastic decoding. 3. Synthetic and trace based load tests: representative prompt/output lengths, arrivals, prefix reuse, concurrency, streaming and GPU/KV pressure; record p95/p99 TTFT and inter token latency. 4. Shadow production traffic: mirror a limited, approved fraction of live requests to a separately capacity provisioned candidate; stable response alone goes to the caller. 5. Analyze paired outputs and performance: compare stable and candidate using correlation IDs, per request input conditions, task quality and inference costs. Account for asynchronous delivery and missing shadow results. 6. Canary production traffic: only after shadow quality/performance gates pass, make the candidate answer a small real user traffic fraction under the multi signal rollback policy from Question 264. 2. Basic architecture A trusted ingress authorizes an incoming user request and sends it to the stable LLM. A sampler chooses whether this request is eligible for mirroring. It must respect tenant consent, data residency, provider, retention and privacy policy . An asynchronous shadow branch sends the same safe request payload and relevant nonsecret configuration metadata to an isolated candidate engine. A telemetry/paired analysis system records request ID, input length class, model/version, prompt template version, stable metrics and candidate metrics. The user receives only the stable model response. Candidate errors or long delays should not increase the user facing critical path latency ; isolate the mirror path with queue, compute and timeout budgets. Important Istio detail: Istio's native traffic mirroring is fire and forget : mirrored responses are discarded, and the shadow branch is out of the primary response path. To compare the candidate answer text , explicitly add consented, secured candidate side response capture and a separate comparator. Traffic mirroring by itself is not an automatic paired output evaluator . 3. Beginner friendly numerical example Suppose production receives 100 requests/second , and engineers choose 10% mirroring : 1. Stable deployment still handles 100 RPS and serves all user responses . 2. Candidate handles approximately 10 shadow RPS ; total fleet model attempts become approximately 110 RPS , assuming successful mirror delivery and no duplicate retries . 3. Over a 10 minute window, expected stable requests = 100 × 600 = 60,000. 4. Expected candidate mirror requests = 60,000 × 0.10 = 6,000. 5. Suppose 5,700 mirrors finish and yield usable candidate records; successful mirror completion fraction = 5,700 / 6,000 = 95%. 6. Of 1,000 independently labeled, paired eligible examples , the stable model answers 930 correctly and the candidate answers 910 correctly: accuracy = 93% versus 91% , a 2 percentage point absolute loss . 7. If the predeclared maximum acceptable degradation is 1 percentage point , do not advance to canary based on this evidence. Analyze confidence, paired disagreement, task slices and evaluation reliability before a final decision. All numbers are illustrative, not measured claims. Mirroring 10% of requests does not necessarily mirror 10% of GPU cost , because selected prompts, output lengths, cache states and candidate model efficiency may differ. 4. Implement sampling and isolation Sample by workload class , not just random requests: include short chat, long RAG, code, structured output, streaming and high KV memory prompts within eligibility policy. Use separate GPU and queue budgets for the candidate; otherwise shadow work can overwhelm production GPU pools, network, object storage or routing. Cap mirror fraction, concurrent mirrors, queued tokens, output length and request lifetime ; gracefully drop mirrors when the shadow system is saturated. Use opaque correlation IDs rather than raw personal identifiers in telemetry. Enforce encryption, least privilege retention, and tenant isolation. Avoid copying credentials, cookies and bearer tokens that permit the shadow deployment to perform user actions. Label shadows explicitly so autoscalers, quotas, billing, alerts and gateway metrics do not incorrectly count them as genuine customer requests. 5. Protect external actions A mirrored prompt may cause the candidate to produce a tool call or agent action. Never let shadow execution send real emails, make purchases, create tickets, call destructive tools or write production data simply because the original user request was authorized. Safe options: Stub or sandbox tools and run with read only test fixtures. Disable side effects at tool dispatch rather than trusting the candidate to refrain from tool calls. Preserve tool schemas and return representative synthetic responses for comparison. Log proposed tool actions for analysis, but do not execute them without separate authorization. Avoid sending private production context to an unapproved candidate model, region or vendor. A shadow result that looks syntactically valid can still be semantically wrong, unauthorized or unsafe. 6. Compare latency fairly Capture stable and candidate first token time, inter token latency, total request time, queue wait, timeout/error rate, input/output tokens and KV memory . Align measurements to each branch's actual start and note mirror dispatch overhead; an asynchronous mirror may start later and can have a different queue or cache state. Compare per length and per tenant/workload slices , not only overall averages. A 10% shadow sample may underrepresent rare long context prompts. Keep model/serving version, hardware, number of GPUs, precision and load profile visible; performance differences from extra candidate capacity should not be attributed solely to the model revision. A standalone shadow pool is excellent for candidate correctness and stability, but it does not automatically prove how the candidate will behave when it replaces the main fleet under full load. Run independent full load benchmarks. If the mirror handler discards streaming responses, measure TTFT/ITL from candidate side tracing, not from the user visible stable response. 7. Compare output quality responsibly Exact match works for fixed format deterministic tasks; it is inappropriate as the sole measure of open ended responses. Compare task success , required JSON/schema validity, tool call argument correctness, groundedness against authorized source documents, domain specific factual checks and policy compliance. Use paired prompts for stable/candidate comparison; blind human judgments or calibrated LLM as judge rubrics may help when no exact label exists. Record decoding temperature, random seeds where supported, system prompt, tools and model revisions. Stochastic differences do not necessarily imply regressions. Separate immediate automated quality measurements from delayed or human labeled outcomes. Shadow testing does not directly measure user preference for unseen candidate answers. 8. Platforms and control planes Istio supports traffic mirroring with a specified mirror percentage; its mirror requests are out of band and mirror responses discarded. Additional capture code is needed for paired response analysis. Amazon SageMaker AI supports production and shadow variants where the production variant serves callers and the shadow variant receives a chosen fraction of duplicated requests. The platform provides comparative metrics and optional data capture, subject to endpoint feature limitations. Argo Rollouts experiments and analysis templates can orchestrate baseline/candidate deployments and evaluate success/failure/inconclusive metrics; traffic duplication and safety guards still need an appropriate router/ingress implementation. Ensure that any mirroring mechanism works with the application's HTTP/SSE/gRPC streaming semantics ; request mirroring of a stream may not preserve full response behavior or lifecycle. 9. Production incident: unsafe shadow experiment Incident: A team mirrors 50% of an agentic chatbot's traffic onto a new model. The shadow endpoint invokes a production ticket creation tool and generates duplicate customer tickets. In parallel, its queue shares a GPU node with stable traffic, raising stable p95 TTFT from 700 ms to 2.1 seconds . Root cause: Shadowing was treated as side effect free, while tool execution and GPU resource isolation were never enforced. The mirror fraction was chosen by request count without checking token work or candidate capacity. Engineering fix: 1. Disable/sandbox all shadow side effecting tools and revoke write capable credentials. 2. Move candidate workloads into a separately bounded compute pool. 3. Lower and stratify mirror sampling; limit concurrency and queued token budgets. 4. Add request/candidate correlation IDs, candidate side response capture and safety/quality comparators. 5. Confirm stable service p95/p99 TTFT, error rate and KV occupancy are unchanged when shadowing is enabled. Prevention: Use a prelaunch shadow safety checklist, test with intentional tool call prompts and resource contention, and enforce automated removal of the mirror route when shadow latency/errors threaten stable serving. 10. Promotion gates and limitations Promote from shadow to canary only when: No critical security, tenant isolation, unauthorized tool or safety regression occurs. Required task accuracy, JSON/tool validity and long context quality pass predeclared thresholds with adequate samples . Candidate TTFT/ITL, errors, GPU cost and KV footprint meet targets under independent realistic load . Stable deployment is not harmed by mirror traffic; shadow sample coverage is sufficient. The rollback and safe stream drain paths from Questions 263 and 264 are operationally tested. Interview ready answer: “I'd run a frozen offline quality suite, then realistic load tests, and only then shadow a privacy approved sample of live requests to a separately isolated candidate. Stable continues answering users; the candidate does not. I'd sandbox tools, collect candidate side responses because a native mirror may discard them, compare paired correctness and TTFT/ITL, and ensure no extra load hits the stable path. After shadow gates pass, I would canary with automated rollback; shadow testing alone does not prove full load or user experience success.” Research references: Istio — Traffic Mirroring Amazon SageMaker AI — Create a Shadow Test Amazon SageMaker AI — Model Validation in Production Argo Rollouts — Analysis and Progressive Delivery
How do you detect and troubleshoot inference performance regressions after model, runtime, or hardware upgrades?
An LLM inference performance regression is a measurable deterioration in serving behavior following a model, runtime, GPU, driver, CUDA, scheduling, or infrastructure change. Typical symptoms include higher time to first token (TTFT) , worse inter token latency (ITL/TPOT) , lower SLO compliant goodput , increased errors, or unexpectedly high cost per successful request. The goal of debugging is to distinguish a true software or hardware regression from traffic differences, changing answer lengths, cache warmness, or faulty measurement. This lesson extends Question 189 (general benchmarking/profiling), Question 258 (realistic traffic and tail latency), and Question 264 (canary rollback) by teaching controlled before/after experiments, change isolation, bottleneck attribution, and regression prevention gates . 1. Start by defining exactly what regressed TTFT: user facing wait before the first token. Break it into gateway/network, queue, tokenization, prefill, KV transfer where relevant, and first decode step. ITL / TPOT: output token cadence; separate decode kernel changes, memory bandwidth/KV reads, synchronization and excessive batch concurrency. Goodput: successful requests meeting all specified TTFT/ITL/error SLOs per second. A higher raw tokens/s result may coexist with worse goodput. GPU memory and KV: bytes available for model and KV, prefix cache hit fraction, preemptions/recomputation, fragmentation and maximum concurrent active sequences. Quality: task accuracy, refusal/format/schema compliance, long context retrieval and model output distribution. Faster but incorrect results are not a successful optimization. Compare p50, p95 and p99, plus request length and tenant/workload slices ; aggregate means can hide a new long prompt failure. 2. Freeze a reproducible comparison Record a deployment fingerprint for both old and new versions: 1. Model weights hash, tokenizer revision, chat template and any LoRA/adapter. 2. Inference engine and compiler/library versions (vLLM, PyTorch, CUDA, NCCL, quantization kernels, accelerator drivers). 3. GPU model, HBM, topology, GPU clocks/power limits, MIG or co tenant placement, tensor/pipeline parallelism. 4. Model/runtime flags: cache dtype and size, prefix caching, max sequence/token budgets, CUDA graphs, compilation/optimization levels, speculative decoding and scheduler policy. 5. Workload: exact prompts, input/output token lengths, arrival schedule, streaming settings, prefix reuse, warmup and request level SLOs. 6. Router/network: replica health/load, routing policy, source region, client generator capacity and request queue. Do not compare a warm old replica with a cold new replica or a 512 token baseline with an 8k token candidate and conclude the runtime regressed. Keep the experiment as close to one variable as practical. 3. Beginner friendly numerical example Assume a matched, illustrative 10 minute load test after an engine upgrade: Old runtime: p95 TTFT = 800 ms , p95 ITL = 35 ms , joint SLO goodput = 12 requests/s . New runtime: p95 TTFT = 1,200 ms , p95 ITL = 42 ms , joint goodput = 9 requests/s . TTFT regression = (1200 − 800) / 800 × 100 = 50%. ITL regression = (42 − 35) / 35 × 100 = 20%. Goodput loss = (12 − 9) / 12 × 100 = 25%. Investigation: The new version's p95 TTFT increase could arise from queueing, slower prefill or a different prompt length distribution. Inspect request level spans before guessing. Suppose controlled tracing finds that p95 queue wait changed from 100 ms to 430 ms , while per request prefill compute increased only slightly. A 330 ms queue delay increase would strongly implicate scheduling/limited capacity, but do not subtract two independent p95 component values and call that the exact p95 total latency contribution; percentiles of components are not generally additive. Join spans for the same requests to identify the causal contribution. These are invented teaching measurements rather than actual benchmark outcomes. 4. A seven step incident debugging workflow Step 1 — Protect users. If latency/error/quality gates fail, stop promoting the candidate and route new requests to the approved baseline; allow existing streams to drain where safe. Capture a small, privacy approved set of traces before rollback if doing so will not worsen the incident. Step 2 — Confirm the change is real. Re run old and new deployments side by side or in a controlled A/B replay. Match arrival rates, request length buckets, output lengths, cache states and hardware; compare confidence intervals and repeat trials. Step 3 — Locate the stage. Use distributed tracing to separate frontend/network, admission queue, tokenization, prefill, optional KV transfer, decode, and streaming flush. Look for the stage accounting for the observed per request increase. Step 4 — Check resource/scheduler symptoms. Compare running/waiting requests, scheduled prefill tokens, KV cache pressure, preemptions, prefix hits, tensor parallel communications, host/GPU transfers and errors. vLLM exposes Prometheus metrics and logging for these. Step 5 — Isolate one change. Re run with an unchanged model and hardware while varying only the runtime; or fix runtime and compare GPU families. Where feasible bisect configuration changes or versions. Avoid toggling compiler, kernel and batching settings simultaneously. Step 6 — Profile a small sample. Use Nsight Systems for GPU timelines/kernel gaps and PyTorch Profiler for richer operator shapes/stacks/memory diagnostics where needed. vLLM cautions that profiling can significantly slow inference— measure final performance in separate unprofiled runs . Step 7 — Validate the fix and prevent recurrence. Repeat matched load sweeps and quality tests, compare p95/p99 TTFT/ITL and goodput, canary under approved traffic, and add the failing workload to CI regression gates. 5. Common regression signatures and root causes TTFT rises, queue wait grows: new scheduler token budgets, reduced ready replicas, slower routing, uneven long prompts, or more preemption reducing service capacity. TTFT rises with stable queue: tokenization/prefill kernel regression, compilation fallback, changed prompt length, lower prefix hit rate or more costly KV transfer. ITL rises, TTFT mostly stable: slower attention/decode kernels, more active decode sequences, GPU HBM bandwidth contention, CPU orchestration gaps or NCCL collectives. GPU memory rises after model change: a larger weight set, changed KV head count/cache dtype, compilation workspace, less efficient allocation or changed max model len. Prefix cache hit rate falls: changed prompt prefix, tokenizer/template, cache hash/eviction settings, router affinity or model compatibility boundaries. Preemptions surge: KV cache too small for concurrency, changed batch size limits or attention layout. In vLLM V1, common preemption uses RECOMPUTE , which increases repeated GPU work. Throughput drops after moving to a newer GPU: mismatched CUDA/kernel builds, lower actual clocks, PCIe/NVLink bottlenecks, parallelism/placement mismatch or an unsupported optimized path. A faster GPU specification does not guarantee a faster whole system . Startup becomes slower but steady state inference is normal: compilation cache miss, extra CUDA graph capture, model weights staging or changed optimization mode. Distinguish cold start latency from ongoing TTFT/ITL. 6. A controlled one factor experiment Suppose p95 ITL worsens after a runtime and driver bundle update. Test safely: 1. Old runtime + old driver (known baseline). 2. New runtime + old driver only if that combination is supported . 3. Old runtime + new driver only if supported . 4. New runtime + new driver (candidate). If a compatibility matrix forbids one combination, do not run unsupported mixes. Use supported intermediate releases or separate matched environments instead. The objective is to attribute the regression to an independent factor or an interaction, not to force an impossible experiment. Keep model, hardware, power settings, exact traffic replay and percentile methodology fixed. Re run with different seeds and realistic workloads. 7. Production incident: new runtime disables an optimized path Incident: A model serving upgrade initially looks healthy in basic readiness checks, but p95 TTFT rises from 0.8 s to 1.2 s and goodput drops from 12 to 9 requests/s after production traffic ramps up. Root cause analysis: Trace comparison reveals queue wait rising on heavily loaded replicas. Scheduler metrics show KV preemptions and recomputation became common for long context requests. The new runtime's cache allocation or concurrency configuration leaves fewer effective KV blocks despite unchanged GPU count. The bottleneck is therefore not necessarily the model weights themselves but changed engine configuration and available KV headroom . Engineering fix: Restore a version compatible cache/concurrency policy; test reasonable adjustments to KV memory allocation and batch token/active sequence limits; verify no OOM; preserve intended prefix cache behavior; retest under realistic mixed input lengths. Apply canary rollout/rollback gates. Prevention: Pin a deployment manifest with full model/runtime/hardware/flags, alert on unexpected preemptions, and require separate cold/warm benchmark plus long context p95/goodput regression tests before promotion. This is a hypothetical incident; the investigation chain is illustrative, not an assertion that a specific vLLM release has this defect. 8. Monitoring and release gates Version labeling: model hash, runtime build, GPU type, attention/quantization backend and rollout cohort for all serving/tracing metrics. Performance: p50/p95/p99 TTFT, ITL, queue wait, output token counts, SLO conforming goodput and actual offered rate. Resource use: vllm:num requests running, vllm:kv cache usage perc, vllm:prefix cache hits, vllm:prefix cache queries, prompt/generation tokens and preemptions where exported. Metric names and availability vary by installed version. Correctness: schema validity, task pass rate, model regression suites and safety policy checks. Reliability: streaming disconnects, 429/503/5xx, latency based admission rejections, cancellation cleanup and worker crashes. Cost: GPU hours per successfully completed SLO conforming request. Noise control: enforce a minimum sample count, stratify by prompt/output length, and use consistent warmup/observation windows. One surprising sample is not proof of a statistically persistent p99 regression. Interview ready answer: “I would reproduce the slowdown under the same model, traffic, GPU and cache conditions, separate queueing from prefill and decode spans, then correlate the affected stage with KV/preemption, kernel, communication and scheduler telemetry. I would isolate changes with supported old/new combinations and use short Nsight/PyTorch traces for cause—but benchmark without profiling to avoid observer overhead. I'd roll back unsafe candidates, verify both latency and answer quality, and add the scenario to automated release gates.” Research references: vLLM — Optimization and Tuning vLLM — Profiling vLLM vLLM — Metrics vLLM — Benchmarking CLI NVIDIA — Nsight Systems
How do you build production observability dashboards and alerts for LLM inference latency, throughput, KV-cache pressure, and errors?
Production LLM inference observability joins user facing service level indicators (SLIs), engine metrics, GPU telemetry, and request traces into dashboards and alerts that explain what is failing, for whom, where, and why . A useful dashboard distinguishes demand , accepted work , successful/SLO compliant completions , queueing , prefill , decode , and KV cache pressure . Alert on customer impact and leading saturation signals, not merely high GPU utilization. This lesson extends Question 189 (profiling fundamentals), Question 257 (goodput calculation), and Question 266 (regression investigation) by focusing on a production observability architecture, PromQL, alert design, runbooks and safe telemetry cardinality . 1. What a production dashboard should answer Is the customer experience healthy? Show p50/p95/p99 time to first token (TTFT) , inter token latency (ITL) or per request time per output token (TPOT) , total latency, and the fraction of original user requests meeting the agreed SLOs. Is demand higher than safe capacity? Plot offered request rate, accepted rate, completed request rate, cancellations/rejections, waiting requests, running sequences and queued prompt tokens. Where is the time going? Split queue wait, prefill, optional KV transfer, decode and response/network overhead using request traces and histograms. Is the model memory constrained? Show fraction of used KV blocks, cache evictions/preemptions, prefix cache hit fraction, GPU HBM and active context length distributions. Are errors increasing? Display 429/503, timeout, inference 5xx, worker restart and stream abort counts separately; classify expected user cancellations rather than treating all as server defects. Is the service efficient? Include prompt/output tokens per second, SLO compliant goodput, cost per successful/SLO conforming request, and availability. 2. Metrics that vLLM exposes Current vLLM v1 Prometheus compatible metrics include, subject to the installed version and feature flags: Running and queued requests: \vllm:num requests running\ and \vllm:num requests waiting\. KV cache pressure: \vllm:kv cache usage perc\ is a 0–1 fraction of used KV blocks (multiply by 100 for percent). Throughput counters: \vllm:prompt tokens total\, \vllm:generation tokens total\ and \vllm:request success total\ (finish reason labels matter). Latency histograms: \vllm:time to first token seconds\, \vllm:inter token latency seconds\, \vllm:request time per output token seconds\, \vllm:e2e request latency seconds\, and request queue/prefill/decode time histograms. Prefix cache counters: \vllm:prefix cache hits\ and \vllm:prefix cache queries\; calculate a ratio over the same time window. Admission rejections: \vllm:admission rejections total\ where supported. GPU metrics: use NVIDIA DCGM exporter or equivalent monitoring for GPU memory, compute, power, temperature and other device signals. Check the actual /metrics payload and engine version before deploying a copied dashboard. Old vLLM versions have different metric names and do not expose every newer metric. For example, legacy GPU cache utilization metrics differ from the current \kv cache usage perc\. Critical distinction: ITL records per stream output gaps; request level TPOT estimates average decoding time per output token per request. They differ when output events contain multiple tokens. Do not label ITL p95 as TPOT p95 interchangeably. 3. Beginner friendly numerical example A fictional five minute window has: 3,000 original requests , of which 2,850 succeed, 90 fail, and 60 are rejected or time out. 2,400 of those original requests finish successfully and meet the joint TTFT and TPOT objectives. A configured target of 99% successful SLO compliant service. Compute: 1. Overall success fraction = \2,850 / 3,000 = 95%\. 2. Joint SLO attainment = \2,400 / 3,000 = 80%\. 3. SLO goodput = \2,400 / 300 = 8 requests/s\. 4. Fraction violating the success plus latency objective = \(3,000 2,400)/3,000 = 20%\. 5. If the permitted failure budget is 1% , the short window error budget burn multiple is \20% / 1% = 20×\. Interpretation: A 20× burn means this window's error budget consumption is twenty times the long term sustainable rate, not that 20% of an entire monthly budget has already been exhausted. A paging policy should require measured volume and appropriate multi window burn rate criteria rather than alerting on one tiny interval. Now suppose \vllm:kv cache usage perc\ rises from 0.65 to 0.96 while waiting requests climb from 3 to 70 and p95 TTFT rises from 0.7 s to 2.5 s . This combination suggests an urgent saturation investigation. High KV usage alone is not proof of a failure: examine preemptions, evictions, active lengths and SLO impact. All numbers are illustrative. 4. A practical metric pipeline 1. Instrument gateway and model engine. Export request outcome counters and histograms from a trusted gateway, and serving/KV metrics from each engine replica. 2. Scrape Prometheus compatible endpoints using ServiceMonitor/PodMonitor or a managed equivalent; verify that labels identify model revision, deployment, region and instance. 3. Create Grafana dashboards with fleet wide customer impact panels at the top and drill down panels for queue, prefill, decode, KV memory and GPU devices. 4. Correlate with OpenTelemetry traces , propagating an opaque request ID across gateway, router, prefill, optional KV transfer, decode and stream output. 5. Use logs for diagnosis , with structured reason codes and version fingerprints, not for storage of raw sensitive prompts by default. 6. Attach runbooks to actionable alerts, including on call severity, likely causes, dashboards, safe mitigations, and escalation. Protect metrics/traces from unnecessary tenant IDs, raw request IDs, prompt contents, bearer tokens and other high cardinality/sensitive labels. Use authorized tracing storage with privacy reviewed retention. 5. Practical PromQL examples The following queries illustrate common vLLM v1 metrics. Confirm names and histogram buckets in the installed version; ensure all required dimensions are consistent. P95 fleet TTFT over five minutes: \\\promql histogram quantile( 0.95, sum by (le) ( rate(vllm:time to first token seconds bucket[5m]) ) ) \\\ P95 TTFT by model name: \\\promql histogram quantile( 0.95, sum by (model name, le) ( rate(vllm:time to first token seconds bucket[5m]) ) ) \\\ Output tokens per second: \\\promql sum(rate(vllm:generation tokens total[5m])) \\\ Current queue size and KV occupancy: \\\promql sum(vllm:num requests waiting) max by (model name) (vllm:kv cache usage perc) \\\ Windowed prefix hit fraction: \\\promql sum(rate(vllm:prefix cache hits[5m])) / clamp min(sum(rate(vllm:prefix cache queries[5m])), 1e 9) \\\ Important: calculate rates from individual counters before aggregation to handle resets. For p95, aggregate histogram buckets before \histogram quantile\; never average pod level p95 values. A Prometheus histogram percentile is an approximation based on its buckets. Define buckets around actual SLO boundaries; too many buckets increase cardinality and storage costs. For business level joint TTFT+TPOT goodput , emit an independent end of request count from an application/gateway evaluator that knows both observations and request success status. You cannot derive a correct joint pass count merely by combining two independent histograms . 6. Alert design: page versus investigate Page on real or imminent customer impact: Multi window, multi burn rate SLO violations at a sufficient sample volume. Sustained high p95/p99 TTFT or TPOT with concurrent increases in queue delay, timeouts or failed requests. Sudden request failure/rejection spike, unavailable serving replicas, or a confirmed severe security/quality issue. Create warning/ticket alerts for leading signals: Sustained queued work and rising oldest request age while offered demand exceeds successful completions. KV occupancy nearing the benchmarked unsafe region plus preemption/eviction or TTFT impact. Falling prefix hit fraction after a new template/router release. Worsening GPU memory fragmentation, model restart rate, GPU hardware errors, cost per good request or startup latency. Avoid noisy alerts: A single high GPU utilization value may be healthy batching. A 99% KV cache gauge may be normal under some workloads; calibrate thresholds against actual preemptions/SLOs. p99 latency at low volume is unstable; enforce a minimum request count threshold and sensible windows. Separate alerts per model/workload/region but avoid one page per replica for one fleet wide incident. Track missing metrics and scrape failures explicitly; missing data is not the same as a healthy zero . 7. Production incident: KV pressure hidden behind aggregate averages Incident: A chat endpoint's fleet average TTFT looks stable, but users with 16k–32k token prompts experience p99 TTFT of 7 seconds , frequent timeouts and poor streaming quality. Debugging steps: 1. Slice TTFT by prompt length buckets, engine revision and replica. Investigate queue time versus prefill time. 2. Observe one replica at 95% KV block use , repeatedly preempting/recomputing long contexts, while others remain at 60% . 3. Inspect prefix reuse, router placement, max model length and per replica batching/active sequence settings. 4. Compare admission rejections and completed request goodput by tenant/workload to see whether errors were masked by success only latency charts. Root cause: The dashboard displayed only fleet average latency and GPU utilization. It hid request length distribution, unbalanced routing, KV/cache pressure and failure exclusions. Engineering fix: Build length sliced p95/p99 and joint goodput panels, add queue/KV/preemption correlations, rebalance eligible work using capacity aware routing, and tune admission/backpressure. Keep paging tied to user impact, not a lone cache gauge. Prevention: Test dashboard/alert coverage using mixed long prompt load, replica failure and forced KV pressure. Require end to end outcome instrumentation as a production release gate. 8. Dashboard structure and runbook A useful first page service health dashboard contains: 1. Request arrivals/accepted/completed/rejected/failed, with error codes. 2. Joint SLO success and error budget burn with p95/p99 TTFT and request level TPOT. 3. Per model/region routing distribution, waiting requests, oldest queue age and active sequences. 4. Prompt/output tokens per second and average/percentile token lengths. 5. KV occupancy, prefix hit fraction, evictions/preemptions and GPU device memory. 6. Canary/stable version comparison and timestamps for deploys, autoscaling events or hardware changes. Example alert runbook: If TTFT suddenly rises, first check waiting queue and admission rate. If queue wait rises with stable prefill, look at capacity, routing, retries and KV preemption. If queue is stable but prefill grows, inspect uncached prompt length, prefix reuse, model/runtime/kernel changes, and GPU communication. If TTFT is stable but TPOT worsens, focus on decode batching, KV bandwidth, GPU contention and collectives. Roll back unsafe releases or shed excess work as the agreed SLO policy permits. 9. Interview ready answer “I'd expose vLLM engine histograms/counters and gateway outcomes through Prometheus, build Grafana panels for p95/p99 TTFT, request level TPOT, queue depth, KV occupancy, token throughput and successful SLO goodput, and join request traces across prefill/decode. I'd use multi window burn rate paging for customer impact, separate leading saturation warnings, and keep labels low cardinality. For debugging, I'd slice by model, prompt length and release version and use a runbook that ties queueing, KV preemption and kernel changes to user visible latency.” References: vLLM — Production Metrics vLLM — Design of Metrics and Grafana Google SRE Workbook — Alerting on SLOs Prometheus — Histograms and Summaries NVIDIA DCGM Exporter
How do you use distributed tracing to diagnose queueing, prefill, KV-transfer, and decode latency in LLM inference?
Distributed tracing follows one LLM inference request across services and GPU workers, recording when each operation starts, how long it takes, and which other operations it depends on . It helps explain why a particular user waited even when fleet wide p95 charts look healthy. An inference trace might cross an API gateway, auth/rate limiter, tokenizer, KV aware router, waiting queue, prefill GPU, optional KV transfer network, decode GPU, and streaming response writer. A metric answers “how much latency?”; a trace helps answer “which operation caused this request's latency?” This lesson continues Question 267 (Prometheus dashboards and alerts), while avoiding a repetition of Question 243 (the mechanics/trade offs of disaggregated prefill/decode). 1. Understand the request timeline For a typical disaggregated LLM: 1. Gateway and preprocessing: authenticate, validate input and apply chat template/tokenization. 2. Router: choose eligible prefill and decode workers, perhaps using prefix cache locality and load. 3. Admission/queueing: wait for a compute slot, token budget and KV cache space. Include router side and engine side queues separately. 4. Prefill: process input prompt tokens and construct the initial KV state. Prefill is often compute intensive for long uncached prompts. 5. KV transfer: move computed KV blocks to a separate decode worker when disaggregated serving is enabled. This stage may include handshake, waiting, DMA and data movement. 6. Decode: iteratively generate new tokens with the evolving KV state. Track first token production and subsequent inter token gaps. 7. Stream delivery: flush output chunks to the gateway/client; network/proxy buffering may add delay even when GPU computation is fast. Not every deployment has each stage. In a colocated engine, prefill and decode share the serving process/GPU and may have no inter worker KV transfer at all. In multimodal serving, vision/audio encoding may introduce another stage. 2. Spans, traces and context propagation A trace is the end to end linked record for a logical request; a span represents an operation such as \inference.prefill\. Each span has a trace ID , span ID, parent relationship, start/end timestamps, status, and safe attributes such as \model.version\, prompt token count, worker class and GPU device type. Use W3C Trace Context (for example, \traceparent\) for HTTP propagation and supported message/RPC headers for internal routing. Preserve trace context through asynchronous scheduling by attaching it to the queued request metadata and restoring the context inside the worker when work begins. If an operation is triggered asynchronously rather than being a nested synchronous call, use OpenTelemetry span links when parent child structure would misrepresent causality. Instrument both sides of important RPC boundaries when possible; otherwise delays can be wrongly blamed on the destination worker. Keep \service.name\, model/version and region attributes consistent, without inserting raw prompts or user secrets into trace tags. OpenTelemetry defines semantic conventions for GenAI inference operations, including latency and token usage. These conventions are evolving; prefer the instrumented library's documented stable/opt in version and add clearly named internal stage spans when needed. 3. Beginner friendly numerical example: one slow request A hypothetical request has the following non overlapping wall clock stages , measured from a single end to end timeline: Gateway validation/tokenization: 20 ms . Router decision: 10 ms . Wait for a prefill slot: 280 ms . Actual prefill compute: 90 ms . KV transfer and handshake: 120 ms . Decode scheduling to first token: 30 ms . First token production and delivery: 50 ms . Total time to first delivered token (TTFT) in this simplified sequential trace: \20 + 10 + 280 + 90 + 120 + 30 + 50 = 600 ms\. Prefill queue's share of TTFT = \280/600 × 100 ≈ 46.7%\. KV handoff's share = \120/600 × 100 = 20%\. Combined queue + KV handoff = \400/600 ≈ 66.7%\. A team may incorrectly optimize attention kernels because the request is “slow on GPU.” The trace instead shows that waiting and handoff dominate first token latency . Now consider the output stream: if the next four token delivery gaps are 40, 45, 110, and 45 ms , the average inter token gap for those observations is \(40+45+110+45)/4 = 60 ms\. The 110 ms outlier warrants a decode scheduler or network flush investigation. The 600 ms TTFT does not include these later delivery gaps. Important: these sums work only because the example deliberately uses non overlapping sequential stages. Real systems overlap CPU work, GPU compute and asynchronous KV DMA. Never add all nested or overlapping span durations and call that end to end latency; examine the critical path , and align spans by request identity and causality. 4. Build the minimum useful trace tree A practical shape is: \\\text HTTP POST /v1/chat/completions (root, gateway) validate and tokenize route request choose prefill worker choose decode worker inference request prefill wait prefill compute kv handoff kv transfer setup kv transfer wait kv transfer bytes decode wait first token ready stream tokens decode step (sampled/aggregated) flush to client \\\ The exact tree varies by serving engine. For example, some systems create multiple worker/RPC spans, or implement a KV transfer overlapping the beginning of decode processing. Use span links if a single sequential parent child tree would hide cross worker timing. Low cardinality attributes: model revision, engine build, route class, prompt length bucket, output length bucket, streaming boolean, cache hit bucket, region, worker class, error type, cancellation reason and sample decision. Prefer a request ID as a searchable trace identifier or log correlation value , not as a Prometheus metric label. 5. An implementation outline with OpenTelemetry Here is minimal illustrative Python instrumentation around a serving pipeline. The imported functions represent your own engine integration; they are not built in vLLM APIs: \\\python from opentelemetry import trace tracer = trace.get tracer("llm serving") async def handle request(req): with tracer.start as current span("llm.inference") as root: root.set attribute("llm.model revision", req.model revision) root.set attribute("llm.input tokens", req.input tokens) with tracer.start as current span("llm.route"): prefill worker, decode worker = await choose workers(req) with tracer.start as current span("llm.prefill.queue"): await prefill worker.wait for slot(req) with tracer.start as current span("llm.prefill.compute"): kv handle = await prefill worker.prefill(req) with tracer.start as current span("llm.kv transfer"): await transfer kv(kv handle, decode worker) with tracer.start as current span("llm.decode"): return await decode worker.generate(req, kv handle) \\\ Production caveat: placing a Python span around an asynchronous CUDA launch may measure only enqueue time , not completed GPU execution. Add engine provided timing or correctly synchronized GPU events where supported; synchronization itself can disturb performance. Likewise, a single \generate\ call may return a stream that completes after the function returns, so wrap the actual stream lifecycle , not only the initial coroutine. Propagate context across processes; in process context alone does not create cross node traces. 6. Trace streaming correctly Record request arrival time , first model token ready time , first chunk received by gateway , first byte/chunk observed at client , and stream finish/cancel/error time where each is observable. TTFT at the client may include gateway, proxy, network and flush delay; model internal first token timing does not. Record stream chunks in a bounded way. Full per token spans on every request can overwhelm collectors and add CPU/GC overhead. Prefer sampled events or aggregates for token gaps, with additional detailed sampling only for anomalous requests. For HTTP server sent events, ensure proxies do not buffer output unintentionally. Mark midstream cancellation and terminal status accurately; a response that started returning tokens but then disconnected is not a successful complete request. 7. Diagnose specific bottlenecks from trace shapes Large prefill queue span: too many long prompts, unfair scheduling, inadequate ready prefill capacity, token budget limits or KV pressure. Check offered token work rate and pending work. Long prefill compute span with small queue: prompt length or cache miss increase, an attention/kernel regression, GPU contention or an unexpected model/config change. Long KV transfer setup/wait: slow handshake, insufficient receive buffers, destination decode queue, backpressure or unavailable KV blocks. Long KV byte transfer span: bandwidth saturation, network topology problems, GPU direct transport fallback, retransmissions or too much uncompressed KV data. Long decode wait/steps: high active sequence count, KV pressure/preemption, poor batch scheduling, attention bandwidth or tensor parallel communication. Fast engine but slow first client chunk: gateway buffering, TLS/network, overloaded reverse proxy or output serialization. One worker disproportionately slow: uneven routing, bad GPU/node health, co tenancy, different build/version or faulty storage/network path. Correlate the trace with Prometheus queue, KV occupancy, preemption, request rate and network/GPU telemetry. A span can show where the delay occurred, but causal attribution may still require a controlled experiment. 8. Tail sampling and overhead Head sampling: choose whether to trace near request entry; low overhead, but may miss rare slow requests. Tail sampling: keep traces after observing outcome/latency; excellent for rare slow, error or timeout cases, but needs memory, buffering and consistent routing through collectors. Consider policies to retain errors, timeouts, slow TTFT, long KV transfers, and a representative healthy baseline sample . Preserve enough examples for long context or low volume tenant classes without storing identifying text. Export tracing asynchronously with bounded buffers and backpressure; never let the collector outage block inference. Use trace to metrics exemplars when available so an engineer can jump from a p99 histogram observation to a representative slow trace. Check for clock skew across nodes. Compare durations from individual monotonic clocks and use synchronization for cross host alignment; raw wall timestamps can produce misleading gaps. 9. Production incident: TTFT spikes despite healthy GPU utilization Incident: A 70B disaggregated model receives long RAG requests. Users report p95 TTFT growing from 650 ms to 1.8 seconds , but prefill GPU utilization and kernel timings appear normal. Debugging: 1. A slow trace shows 200 ms prefill compute , 150 ms admission wait , but 1.1 seconds in KV handoff . 2. Child spans reveal the majority of KV handoff delay is waiting for a decode destination to accept cache blocks , not actual network byte transfer. 3. Decode worker metrics show high KV occupancy, pending requests and repeated preemptions; the router sends traffic disproportionately to one group. 4. Compare cache transfer metrics and worker pool versions to rule out a network problem before increasing link bandwidth. Root cause: Decode side KV capacity/backpressure causes a handoff bottleneck, even though prefill GPUs remain busy and look healthy. Engineering fix: rebalance compatible decode workers, tune admission/concurrency to KV capacity, scale decode capacity if needed and avoid routing to saturated workers. Verify that any disaggregated KV state remains correctly owned/isolated during retries or cancellations. Prevention: instrument handoff wait separately from bytes on wire transfer , alert on KV occupancy with rising queue/TTFT, and replay mixed long prompt burst traffic before rollout. This scenario is illustrative, not a claim about an actual production incident. 10. Privacy, security and common mistakes Do not store raw prompts, outputs, retrieved private documents, authorization tokens, user IDs or full tool arguments in traces by default. Use explicit privacy review, appropriate redaction, access controls and short retention for any exceptional content capture. Keep trace context untrusted at public ingress : do not use client provided baggage or trace IDs for authorization or cost attribution; sanitize them and apply sampling/rate limits to prevent telemetry abuse. Distinguish model token ready from user observed first token , as proxies and network add latency. Never blindly add nested spans or sum overlapping KV DMA and decode time. Don't use total request duration for requests that are still streaming or omit rejected/cancelled ones from operational reporting. Confirm vLLM/Dynamo/OTel version specific instrumentation rather than assuming all stage spans exist automatically. Keep a controlled no tracing benchmark: profiling and excessively detailed traces can change scheduling and tail latency. 11. Interview ready answer “I'd propagate OpenTelemetry context from the ingress through router, queues, prefill workers, KV handoff, decode workers and streaming response. I'd record queue wait, compute, KV transfer setup versus bytes on wire, first token ready and client visible TTFT as separate operations. For slow requests I'd inspect the critical path and correlate spans with KV occupancy, queue and network metrics, then validate with a controlled load test. I'd use bounded sampling and privacy safe attributes, and avoid assuming GPU launch spans represent completed execution.” References: NVIDIA Dynamo — Request Flow and Architecture NVIDIA Dynamo — Disaggregated Serving OpenTelemetry — Trace Semantic Conventions OpenTelemetry — GenAI Client Inference Conventions OpenTelemetry — Context Propagation
How do you diagnose GPU memory leaks, KV-cache fragmentation, and long-running inference worker instability?
Long running LLM inference workers can become unstable because of genuine memory leaks, legitimate KV cache growth under load, allocator fragmentation, memory held by CUDA graphs, or problems outside GPU memory entirely. The first task is to classify the failure , not assume that every high memory reading is a leak. A true leak means memory remains reachable or otherwise cannot be reclaimed after the workload that required it has finished. Caching intentionally holds reusable allocations; fragmentation leaves free capacity in pieces or pools that cannot satisfy a required allocation; KV cache pressure means too many active token contexts compete for a finite serving cache. Those situations demand different fixes. This lesson follows Question 267 (dashboard level monitoring) and Question 268 (per request tracing), concentrating on day 2 memory forensics, soak test reproduction, OOM response, and worker recovery . It does not repeat the purpose of KV caching or the basics of prefix cache eviction. 1. First distinguish five failure modes Expected steady state allocation: model weights, kernels, CUDA contexts, CUDA graph private pools, workspace allocations and a reserved KV cache pool may occupy GPU memory while the service is healthy and idle. High utilization alone is not a leak. Dynamic KV cache pressure: active sequences and their prompt/output lengths consume KV blocks. When workload concurrency or context length rises, the cache fills, new requests wait and the scheduler may preempt/recompute work. Memory should return to the pool as requests finish, although the pool may remain allocated to the worker. PyTorch caching allocator: memory reserved by PyTorch may stay above memory allocated to live tensors. The reserved but unused portion is not necessarily leaked and can be reused later. Fragmentation: there may be sufficient total unoccupied or cached bytes but no suitable allocatable block or usable private pool capacity for a request. CUDA graph and eager allocations may be restricted to distinct pools. True retention/leak or worker fault: tensors, request states, references, buffers, native allocations, file descriptors, pinned host memory, threads or failing GPU/device/runtime components accumulate after completed requests. A crashed worker can also result from CPU RAM exhaustion, driver faults, network/device errors, deadlocks or an external termination signal. A restart alone does not prove a GPU memory leak . 2. What to measure at three levels GPU device / process: GPU total/free/used memory, per process memory, GPU memory errors, temperature/power and worker restarts. NVIDIA DCGM or NVML can observe device/process consumption that PyTorch's allocator does not control. PyTorch/CUDA allocator (where accessible in the worker): \torch.cuda.memory allocated()\: bytes currently held by live tensors tracked by PyTorch. \torch.cuda.memory reserved()\: bytes retained by the PyTorch caching allocator. \torch.cuda.memory stats()\: allocator statistics including active, reserved and inactive split bytes; interpretation depends on the active allocator backend. \torch.cuda.memory summary()\ and targeted memory snapshots : allocation history, OOM events, block splits and allocation stack traces. Important visibility limit: PyTorch's memory profiler cannot see all CUDA allocations made directly by custom kernels, third party libraries or other processes. Compare against NVML/device memory. LLM engine: KV cache blocks in use and available, queue depth, active sequences, cache evictions/preemptions, prompt/output token distributions, request completion and cancellations. Engine counters such as \vllm:kv cache usage perc\ and preemption totals are not equivalent to process GPU memory percentage. Model/adapter version, engine configuration, KV dtype, maximum context length and CUDA graph capture settings. 3. Beginner friendly numerical example: leak or caching? Assume an 80 GiB GPU runs a model serving process. All figures are illustrative and refer to a controlled sample after requests have completed : PyTorch live allocated memory: 42 GiB . PyTorch reserved memory: 60 GiB . NVML reported process/device usage (simplified): 66 GiB . Calculate: 1. Allocator managed reserved minus live allocated = \60 − 42 = 18 GiB\. Some of this may be reusable cached memory; do not immediately call it an 18 GiB leak. 2. Approximate usage outside the reported PyTorch reserved amount = \66 − 60 = 6 GiB\. This may include CUDA contexts, native/third party allocations and other memory, not necessarily a leak. 3. Nominal device free memory = \80 − 66 = 14 GiB\, assuming no other consumers and ignoring accounting differences. Now rerun the same fixed workload , allow requests to finish, and compare snapshots at the same quiet point: Hour 1: live allocated 42 GiB . Hour 2: live allocated 44 GiB . Hour 3: live allocated 46 GiB . Hour 4: live allocated 48 GiB . Persistent +2 GiB per hour after equivalent work and cleanup raises suspicion of retained tensors or references. It does not alone prove a leak: verify whether KV usage, CUDA graphs, request length, compilation warmup, adapter caching or allocation pools are still legitimately growing. If the excess rises at the same rate from 42 to 48 GiB, a naive linear forecast would suggest ~8 further hours to reach 64 GiB allocated . Such extrapolation is only a risk indicator: actual OOM timing depends on other allocations, fragmentation, traffic and the memory budget. 4. Reproduce with a disciplined soak test 1. Freeze conditions: model/weights/tokenizer, runtime build, GPU type, engine flags, quantization, CUDA graphs, scheduler, adapters and routing. 2. Warm up deliberately: let compilation, CUDA graph capture, expected cache growth and dynamic allocations settle before measuring trends. 3. Use a bounded repeated workload: reproduce the same prompt/output length distributions, concurrency, cancellations and streaming behavior across many cycles. 4. Collect time series: device memory, per process allocated/reserved memory, host RAM, open files, KV occupancy, active and waiting requests, preemptions, p95 TTFT/ITL and failures. 5. Mark quiet checkpoints: wait for active/queued requests to drain; compare memory at equivalent workload states, not one idle and one busy. 6. Trigger unusual paths: long context prompts, client disconnects, timeouts, tool/schema errors, adapter swapping and model revision rollover. 7. Repeat with a known good version: compare the memory slope and failure count rather than relying on a single run. 8. Capture snapshots near the failure , preferably in a controlled replica. Keep profiler overhead and memory snapshot retention bounded. A high watermark that reaches a plateau after warmup is often a healthy cache . A monotonically rising live allocation baseline across repeated identical completed workloads warrants deeper retention analysis. 5. Diagnose fragmentation with allocator evidence Inspect \memory reserved\, \memory allocated\, inactive split stats, allocation size distribution and allocation history snapshots. Check if allocations are trapped in CUDA graph private pools or fragmented into pieces that the requesting allocator cannot reuse. Compare OOM logs with actual process/device free memory and allocation sizes. A GPU with free bytes in one pool can fail an allocation requiring another pool. Distinguish KV block allocation constraints inside the inference engine from PyTorch's general CUDA heap fragmentation . vLLM's paged KV cache design mitigates traditional variable length contiguous KV waste but does not eliminate block granularity overhead, capacity pressure or unrelated allocator fragmentation. PyTorch \empty cache()\ releases unused cached blocks from its allocator for other applications; it does not free live tensors , cannot resolve retained references, and is not a blanket production fix. Inference engines may use other memory managers. \PYTORCH ALLOC CONF=max split size mb:...\ may help some native allocator fragmentation situations but carries performance trade offs. PyTorch documents this as a last resort when inactive split blocks are unusually high; the setting is not meaningful for \backend:cudaMallocAsync\. Test allocator changes against equivalent latency, throughput and quality objectives before rollout. 6. Handle KV cache exhaustion differently Symptoms: KV cache usage approaches the configured limit during high concurrency or long prompts. Waiting requests and preemption/recomputation counts increase. TTFT and end to end latency deteriorate despite the model weights being loaded correctly. Engineering actions: 1. Check workload mix, prompt lengths, decode lengths and concurrency; ensure the regression is not just a demand spike. 2. Bound admission, active sequences and token budgets appropriately; for vLLM, settings may include \max num seqs\ and \max num batched tokens\. 3. Increase the safe KV cache allocation only when verified physical GPU memory headroom is available. Raising \gpu memory utilization\ without accounting for other allocations can trigger OOM. 4. Scale or reroute eligible long context requests; maintain latency fairness for smaller prompts. 5. Consider a validated KV dtype/quantization strategy, longer context allocation design or more suitable parallel topology, with answer quality and cost checks. 6. Verify that cancellations and finished requests free their logical KV blocks promptly. A stale reference to completed sequences is a bug; a large preallocated pool is not. vLLM explains that insufficient KV cache can cause RECOMPUTE preemption and increased latency; tuning concurrency and cache capacity helps only within the real hardware budget. 7. Practical Python diagnostics Run this inside a controlled PyTorch GPU worker that actually owns the allocations, not from an unrelated gateway process: \\\python import torch if torch.cuda.is available(): dev = torch.cuda.current device() allocated = torch.cuda.memory allocated(dev) reserved = torch.cuda.memory reserved(dev) free, total = torch.cuda.mem get info(dev) stats = torch.cuda.memory stats(dev) gib = 1024 3 print(f"Allocated: {allocated/gib:.2f} GiB") print(f"Reserved: {reserved/gib:.2f} GiB") print(f"Device free: {free/gib:.2f} GiB") print(f"Device total: {total/gib:.2f} GiB") print(f"Inactive split: {stats.get('inactive split bytes.all.current', 0)/gib:.2f} GiB") print(torch.cuda.memory summary(dev, abbreviated=True)) \\\ Interpretation: \mem get info\ is device wide, whereas \memory allocated\ and \memory reserved\ are PyTorch allocator views for the current process. They will not add up cleanly if multiple processes or native CUDA allocations exist. Backend specific memory stats can be unavailable or zero. A full snapshot or history capture may consume extra RAM and add performance overhead. For controlled reproductions, PyTorch documents a memory snapshot and visualization workflow: \\\python torch.cuda.memory. record memory history(max entries=100000) run the bounded test workload torch.cuda.memory. dump snapshot("gpu memory snapshot.pickle") \\\ The leading underscore snapshot functions are private APIs and may change; consult the installed PyTorch version. Snapshots may contain sensitive allocation stack information and must not be uploaded to public services without approval. 8. Production incident: worker restarts every few hours Incident: A vLLM deployment successfully serves daytime requests but starts throwing CUDA OOM errors after six hours. Operators see 95% GPU memory usage and assume the KV cache leaked. Investigation: 1. Compare failed and healthy workers under the same model and trace replayed workload. 2. After active requests drain, PyTorch live allocation memory continues rising by ~2 GiB/hour on the affected build; KV block use returns to its normal baseline. 3. Memory snapshots reveal request scoped tensors retained in an application side response debugging list after cancelled streaming sessions. 4. The GPU also has reserved cache memory, but that stable reserved allocation was not the growing retained allocation . 5. Replaying long streaming requests plus disconnects reproduces the fault; a known good version does not. Root cause: A request lifecycle cleanup bug holds GPU references after cancellations. It is a true retention issue , not expected KV cache residency or fragmentation. Fix: Remove the long lived references, enforce \try/finally\ cleanup for cancelled streams, release engine owned request/KV handles, and add a bounded debug record buffer containing safe CPU only metadata. Test cancelled, timed out and failed requests under soak load; confirm no upward live allocation trend at equivalent quiet checkpoints. Temporarily restore service health by replacing affected replicas through graceful draining , but recognize that periodic restarts only mask the root cause. Prevention: Add memory slope and cleanup invariants to worker soak test gates, alert on unexpected growth together with OOM/latency symptoms, and keep release/version identifiers for root cause correlation. This is an illustrative incident, not a report of an actual vLLM defect. 9. Recover without creating an outage Contain: stop routing new requests to the unstable replica, maintain sufficient ready capacity, and record failed/cancelled streams accurately. Drain safely: complete existing work when possible and enforce bounded termination deadlines. Never assume a streaming generation can resume on an arbitrary new worker with no state transfer. Restart/replace selectively: use staggered rollouts and warm replacements so a fleet wide crash loop does not compound model download pressure. Avoid restart storms: cap retries, add backoff, investigate repeated OOM causes, and ensure admission controls protect the remaining pool. Use safe circuit breakers: prevent a degraded worker from receiving new long context workloads while repairs are tested. Investigate non memory causes: host RAM, CPU thread/file descriptor leaks, device ECC/Xid errors, collective deadlocks, network transport faults, kernel/driver compatibility and worker health probes. 10. Alerts, trade offs and acceptance criteria Track: GPU device memory, allocator live/reserved/inactive split, KV occupancy and preemption rates. Running/waiting sequences, prompt/output length distribution, cancellation cleanup and worker restart/OOM counts. Host RSS/pinned memory, open file descriptors, GPU hardware errors, p95/p99 TTFT/ITL and successful SLO goodput. The slope of live allocated memory at comparable quiet checkpoints , not just absolute vidia smi\ usage. Avoid: Paging on high but stable reserved GPU memory without failures. Calling every allocation failure “fragmentation” without snapshots. Increasing KV allocation to the point that weight/graph/workspace memory no longer fits. Using \empty cache()\ as a cure for a real leak or repeatedly restarting the entire fleet during load. Treating PyTorch only snapshots as complete visibility into custom engine allocations. Release gate: A candidate should survive representative load, cancellation and soak scenarios with no unexplained retained memory growth, stable KV return to pool behavior, no GPU OOM/crash loops, and acceptable TTFT/ITL/quality. Document the runtime/allocator configuration because results depend on it. Interview ready answer: “I would first separate device memory, PyTorch live allocations, reserved allocator memory, and engine KV block usage. Then I would reproduce the fault with identical load and quiet checkpoints, inspect inactive split blocks and snapshots, and check preemptions and cancellations. A steadily rising live allocation baseline indicates retention; stable reserved memory may be caching; high KV use with preemptions means capacity pressure. I'd fix the relevant owner, drain unhealthy replicas safely and add soak test cleanup gates.” Research references: vLLM — Optimization and Tuning: Preemption PyTorch — Understanding CUDA Memory Usage PyTorch — CUDA Memory Management PyTorch — CUDA Memory Statistics NVIDIA — CUDA Graph Memory Issues
How do you diagnose and recover from GPU OOM crashes, CUDA errors, and unhealthy LLM inference workers in production?
Production inference recovery starts by identifying the failure class, isolating the affected worker or GPU, protecting active requests, and replacing capacity safely. A CUDA out of memory (OOM) error, an illegal memory access kernel failure, a failed tensor parallel collective, and a hardware Xid alert can look similar to an end user, but they do not have the same safe recovery procedure . Treat availability, SLOs, and data integrity as first class outcomes—not merely whether a restarted process becomes reachable. This lesson builds on Question 269 (leaks and fragmentation), Question 263 (stream safe draining), and Question 255 (general failover). Its focus is incident classification, node/worker health, crash containment, safe recovery, and operational playbooks rather than memory forensics or traffic failover in general. 1. Classify the failure before retrying Recoverable capacity or admission problem: a request arrives when the engine has insufficient KV blocks, active sequence slots, or memory. If the serving engine supports safe admission refusal, reject/queue within the deadline instead of launching a doomed allocation. A raised PyTorch OOM may be recoverable in some configurations only if the engine explicitly supports the recovery path and its CUDA state remains valid. Process level CUDA failure: illegal memory access, device side assertion, failed CUDA graph launch, or certain CUDA runtime errors may leave the process's CUDA context unusable. The safest default is to mark that worker unhealthy and replace its process , rather than catch and continue without validation. GPU/driver hardware fault: NVIDIA Xid, GPU lost/unavailable, repeated ECC errors, or driver reset can require quarantining the affected device or node . Restarting one pod on the same faulty GPU might reproduce the incident. Distributed group failure: a single failed tensor parallel rank, NCCL communicator, or unavailable KV transfer peer can stall multiple otherwise healthy workers. Recover or replace the whole consistency/parallelism group as required by the serving engine. False positive health failure: a slow model loading operation, kernel compilation, CPU pause, or network scrape delay can fail an aggressive liveness check. Distinguish genuine unresponsiveness from expected startup work. Operational rule: If the CUDA context may be corrupted or device health is uncertain, do not resume serving on that worker simply because the next Python exception handler returns . 2. A five minute triage checklist 1. Identify impact: offered requests versus successful completions, timeouts, 5xx/429/503, stream truncations, p95 TTFT and TPOT, SLO goodput, and how many compatible replicas remain. 2. Identify the fault domain: one request, model process, pod, GPU, Kubernetes node, tensor parallel group, or storage/network dependency. 3. Collect logs and metadata: first CUDA error (not just the last propagated exception), model/runtime revision, GPU/driver/CUDA versions, engine settings, prompt length class, KV cache use, preemptions and restart events. 4. Check cluster state: pod exit reason/OOMKilled (host RAM), GPU allocatable count, pending pods, NVIDIA device plugin/GPU Operator logs, and Xid/kernel events. 5. Contain: stop new routing to the affected worker or group; allow safe active streams to drain if the engine can still serve them. Otherwise terminate failed streams with explicit errors and bounded retry behavior. 6. Replace with a known good model/runtime on a verified healthy GPU; verify readiness by actually exercising the intended inference path. 7. Decide whether it is a local incident or a widespread regression, and whether to roll back the new release or escalate to GPU/driver repair. Do not log raw private prompts, credentials or retrieved documents during debugging without a specific authorized retention plan. 3. Beginner friendly numerical example: capacity after a crash Suppose a 4 replica fleet handles a workload at 6 SLO compliant requests/s per healthy replica . Production offered demand is 18 requests/s . Normal benchmark safe capacity = \4 × 6 = 24 requests/s\. One replica crashes: remaining capacity = \3 × 6 = 18 requests/s\. Theoretical spare headroom is now zero ; burstiness and long prompts may violate SLOs even though nominal capacity equals arrival rate. A second replica fails: \2 × 6 = 12 requests/s\, leaving a 6 requests/s deficit . If demand persists and requests are neither rejected, timed out, nor cancelled, an unbounded waiting queue can grow by roughly \6 × 60 = 360 requests\ in one minute. An appropriate recovery is not “retry all 360 immediately.” The gateway needs bounded admission, per request deadlines, fallback policy, and new ready replica capacity. Choose a protected operating limit below benchmark saturation and re evaluate N−1 resilience. Memory example: An 80 GiB GPU with 72 GiB already consumed and a new 12 GiB allocation request lacks sufficient nominal free capacity (\80 − 72 = 8 GiB\), even before fragmentation/overheads. Lowering batch/KV demand or scaling out may prevent repeats; blindly increasing memory utilization flags cannot create physical VRAM. All numbers are hypothetical teaching examples. 4. Distinguish GPU OOM from host OOM CUDA OOM: PyTorch or engine logs show failed device allocation; the process may remain alive, depending on failure handling. GPU memory, KV pressure and CUDA allocations are the relevant diagnostics. Linux/Kubernetes OOMKilled: container exits with memory pressure related to host RAM/cgroup limits , potentially including CPU side checkpoint buffers, tokenization, pinned host memory or KV offload—not automatically GPU HBM exhaustion. Startup OOM: loading model weights, CUDA graph capture or workspace allocation exceeds available GPU memory even with zero requests. Check dtype, tensor parallel size, context/KV reservation, quantization, CUDA graphs and foreign processes. Steady state OOM: active generations, contexts, allocation fragmentation or a real leak push memory over the limit during traffic; follow the quiet checkpoint and snapshot workflow from Question 269. Runtime/kernel fault: an “illegal memory access” may occur with ample free VRAM, so diagnosing it as KV cache shortage is incorrect. 5. How to debug CUDA errors safely Use an isolated reproduction , not broad high overhead debug flags on a customer facing fleet: Start from the first failing CUDA call and its model/request/worker context. CUDA execution is asynchronous, so a later API call might surface an earlier kernel fault. vLLM documents \CUDA LAUNCH BLOCKING=1\ for localizing the failing CUDA kernel. It changes execution timing and harms performance; use it for controlled debugging only. When a crash appears near CUDA Graph replay, vLLM documents \ enforce eager\ to disable graphs temporarily and better isolate the faulty operation. Use NVIDIA Compute Sanitizer or equivalent kernel debugging for reproducible illegal access or race conditions when appropriate; expect substantial overhead. For multi GPU failures, inspect NCCL logs, rank status, topology, driver compatibility and collectives; diagnose whether one rank died before others reported timeouts. A version/driver/PTX incompatibility can prevent loading or crash kernels. Compare the supported runtime/CUDA/driver matrix and validate with a small compatible baseline. Recoverability warning: Some allocation failures can be caught and handled by a serving framework, but illegal memory access and device side assertions should normally trigger process replacement rather than silently continuing with a potentially invalid context. 6. GPU Xid alerts and device quarantine NVIDIA Xid codes are generated by the driver to report GPU events. Not every Xid has the same severity , so inspect the code and official guidance: Check the affected GPU UUID, timestamp and kernel log (\dmesg\/journald), alongside GPU Operator or device plugin logs. NVIDIA GPU Operator troubleshooting documents a device plugin marking GPUs unhealthy for certain critical Xid errors, reducing Kubernetes allocatable GPU capacity. Where configured, GPU health management may mark a GPU lost or tainted and prevent new work from scheduling to it. Quarantine suspect devices/nodes on critical errors; replace capacity on other healthy hardware . Follow vendor/cloud procedures for reset, drain, maintenance or node replacement. Do not repeatedly clear/restart pods on a device reporting persistent serious Xid/ECC faults. Escalate to SRE/cloud/hardware support. Kubernetes caveat: mark or drain the correct node/device using your operator's procedures; a generic \kubectl drain\ does not magically migrate running GPU memory or fix a damaged GPU. 7. Recovery workflow for a production deployment 1. Detect and isolate: readiness/health controller withdraws failed engine groups from new traffic. The gateway should not route by container liveness alone. 2. Preserve a healthy pool: protect stable replicas from retry storms using rate limits, exponential backoff with jitter, request deadlines and admission control. 3. Replace safely: schedule a new matching model/adapter/runtime replica on a verified healthy GPU or approved fallback hardware. 4. Warm and validate: wait for weights, graph compilation, KV reservation and a small authorized inference readiness probe. Do not count merely scheduled or process running replicas as ready serving capacity. 5. Reconnect routing: add the replacement gradually; confirm queue age, TTFT/ITL, completion/error rates, and the fraction of original requests satisfying the SLO. 6. Quarantine bad nodes: hold critical Xid/driver failure devices out of service until cleared with operator/vendor guidance. 7. Run a controlled reproduction: preserve approved crash evidence and determine whether the fault follows requests, software version, GPU device, or workload pattern. 8. Rollback if systemic: if faults follow a new engine/model/hardware configuration, stop its rollout and return to known good compatible versions. 9. Close with a prevention test: add the initiating failure path to load, cancellation, cold start, failure injection or soak suites as appropriate. A streaming response interrupted by process death generally cannot be resumed at the next token on an arbitrary new worker. Explicitly distinguish client retry/restart from true stateful continuation, and protect tool side effects from duplication. 8. Health probes and crash loop control Startup probe: allow potentially long model checkpoint loading, CUDA graph capture and graph/warmup initialization before applying strict liveness checks. Readiness probe: require a working compatible inference engine, not merely an open HTTP port. On an unhealthy event, withdraw new traffic quickly. Liveness probe: restart genuinely deadlocked/unrecoverable processes; avoid cycling workers that are simply under momentary high load. Termination grace: let healthy in flight streams drain when feasible; no amount of grace fixes a dead CUDA context. Restart backoff: avoid repeatedly spawning cold model workers simultaneously; uncontrolled checkpoint downloads can overload storage and inflate recovery time. GPU group atomicity: a tensor parallel group may need coordinated teardown/rebuild if a rank or NCCL communicator is unrecoverable. Kubernetes and serving controller health definitions vary; test the actual probes, model loading, routing behavior and failure handling rather than relying on defaults. 9. Production incident: two failures with different fixes Incident A — expected high concurrency GPU OOM A chat model starts OOMing when multiple long context prompts arrive together. Before failures, KV occupancy and active sequence count rise; no serious GPU Xid errors appear. Restarting clears active allocations, but the issue returns under the same load. Root cause: Serving concurrency and KV demand exceed a safe memory envelope. Fix: reduce admitted token work or concurrent sequences, select a supported KV allocation configuration, add GPU capacity, and load test the long context workload. Verify that cancelled requests free logical KV blocks. Only treat continued memory growth at equivalent quiet checkpoints as a possible leak. Incident B — GPU fault masquerading as a crash loop After a driver change, one node reports a critical Xid and repeated model worker failures. Pods rescheduled to the same GPU immediately fail even at light load. Root cause: Device/driver fault domain rather than heavy inference traffic. Fix: isolate the faulty GPU/node, bring up matching replicas elsewhere, use NVIDIA operator Xid diagnostics, and involve the hardware/cloud team. Do not mask the issue by endlessly increasing pod restart count. 10. Operational metrics and guardrails Monitor: Customer impact: p95/p99 TTFT and TPOT, stream aborts, timeouts, request acceptance and SLO compliant goodput . Worker health: engine uptime, restart count, last fatal error category, ready versus desired replicas, load times and failures during warmup. GPU faults: Xid by code/GPU UUID, ECC/device health, allocated versus expected GPU count, driver/runtime version. Memory: GPU device free/used HBM, PyTorch allocated/reserved where applicable, KV cache occupancy, preemptions, host RSS and container OOMKilled events. Distributed runtime: failed NCCL collectives, rank exits, prefill/decode/KV transfer disconnects and recovery time. Retries: incoming original requests versus attempted downstream executions; a retry storm can produce high GPU load while success remains low. Do not: Treat a container restart as evidence of successful recovery before verifying healthy inference and SLOs. Catch and continue after an unrecoverable CUDA error without a documented safe reset/recovery path. Keep scheduling workloads on GPUs marked unhealthy by the device plugin/operator. Restart an entire healthy fleet because one replica's KV memory is exhausted. Respond to every OOM by increasing GPU memory utilization settings. Retry streamed tool/action requests blindly; incomplete output and side effects need explicit idempotency/recovery rules. 11. Interview ready answer “I would first determine whether the failure is GPU OOM, host OOMKilled, a process level CUDA fault, an NCCL group failure or a hardware Xid. I'd isolate affected routing, protect healthy replicas with bounded queues and retries, then replace the failed process or group on verified healthy hardware. For OOM I'd examine KV pressure, contexts and allocations; for illegal memory access I'd reproduce with launch blocking/eager mode in isolation; for critical Xids I'd quarantine the device. I'd verify real inference readiness, p95 TTFT/ITL and SLO goodput and add regression/failure tests.” Research references: vLLM — Troubleshooting CUDA and Distributed Serving NVIDIA GPU Operator — Troubleshooting Xid and Unhealthy GPUs NVIDIA DRA Driver — GPU Health Checking PyTorch — CUDA Memory Management and OOM Kubernetes — Configure Liveness, Readiness and Startup Probes