How do you detect and troubleshoot inference performance regressions after model, runtime, or hardware upgrades?

An LLM inference performance regression is a measurable deterioration in serving behavior following a model, runtime, GPU, driver, CUDA, scheduling, or infrastructure change. Typical symptoms include higher time to first token (TTFT), worse inter-token latency (ITL/TPOT), lower SLO-compliant goodput, increased errors, or unexpectedly high cost per successful request. The goal of debugging is to distinguish a *true software or hardware regression* from traffic differences, changing answer lengths, cache warmness, or faulty measurement.

This lesson extends Question 189 (general benchmarking/profiling), Question 258 (realistic traffic and tail latency), and Question 264 (canary rollback) by teaching controlled before/after experiments, change isolation, bottleneck attribution, and regression-prevention gates.

1. Start by defining exactly what regressed

• TTFT: user-facing wait before the first token. Break it into gateway/network, queue, tokenization, prefill, KV transfer where relevant, and first decode step.
• ITL / TPOT: output-token cadence; separate decode kernel changes, memory bandwidth/KV reads, synchronization and excessive batch concurrency.
• Goodput: successful requests meeting *all* specified TTFT/ITL/error SLOs per second. A higher raw tokens/s result may coexist with worse goodput.
• GPU memory and KV: bytes available for model and KV, prefix-cache hit fraction, preemptions/recomputation, fragmentation and maximum concurrent active sequences.
• Quality: task accuracy, refusal/format/schema compliance, long-context retrieval and model-output distribution. Faster but incorrect results are not a successful optimization.
• Compare p50, p95 and p99, plus request length and tenant/workload slices; aggregate means can hide a new long-prompt failure.

2. Freeze a reproducible comparison

Record a deployment fingerprint for both old and new versions:

1. Model weights hash, tokenizer revision, chat template and any LoRA/adapter.
2. Inference engine and compiler/library versions (vLLM, PyTorch, CUDA, NCCL, quantization kernels, accelerator drivers).
3. GPU model, HBM, topology, GPU clocks/power limits, MIG or co-tenant placement, tensor/pipeline parallelism.
4. Model/runtime flags: cache dtype and size, prefix caching, max sequence/token budgets, CUDA graphs, compilation/optimization levels, speculative decoding and scheduler policy.
5. Workload: exact prompts, input/output token lengths, arrival schedule, streaming settings, prefix reuse, warmup and request-level SLOs.
6. Router/network: replica health/load, routing policy, source region, client generator capacity and request queue.

Do not compare a warm old replica with a cold new replica or a 512-token baseline with an 8k-token candidate and conclude the runtime regressed. Keep the experiment as close to one variable as practical.

3. Beginner-friendly numerical example

Assume a matched, illustrative 10-minute load test after an engine upgrade:

• Old runtime: p95 TTFT = 800 ms, p95 ITL = 35 ms, joint SLO-goodput = 12 requests/s.
• New runtime: p95 TTFT = 1,200 ms, p95 ITL = 42 ms, joint goodput = 9 requests/s.
• TTFT regression = (1200 − 800) / 800 × 100 = 50%.
• ITL regression = (42 − 35) / 35 × 100 = 20%.
• Goodput loss = (12 − 9) / 12 × 100 = 25%.

Investigation: The new version's p95 TTFT increase could arise from queueing, slower prefill or a different prompt length distribution. Inspect request-level spans before guessing.

Suppose controlled tracing finds that p95 queue wait changed from 100 ms to 430 ms, while per-request prefill compute increased only slightly. A 330-ms queue-delay increase would strongly implicate scheduling/limited capacity, but do not subtract two independent p95 component values and call that the exact p95 total-latency contribution; percentiles of components are not generally additive. Join spans for the *same requests* to identify the causal contribution.

These are invented teaching measurements rather than actual benchmark outcomes.

4. A seven-step incident debugging workflow

Step 1 — Protect users. If latency/error/quality gates fail, stop promoting the candidate and route new requests to the approved baseline; allow existing streams to drain where safe. Capture a small, privacy-approved set of traces before rollback if doing so will not worsen the incident.

Step 2 — Confirm the change is real. Re-run old and new deployments side by side or in a controlled A/B replay. Match arrival rates, request length buckets, output lengths, cache states and hardware; compare confidence intervals and repeat trials.

Step 3 — Locate the stage. Use distributed tracing to separate frontend/network, admission queue, tokenization, prefill, optional KV transfer, decode, and streaming flush. Look for the stage accounting for the *observed per-request* increase.

Step 4 — Check resource/scheduler symptoms. Compare running/waiting requests, scheduled prefill tokens, KV-cache pressure, preemptions, prefix hits, tensor parallel communications, host/GPU transfers and errors. vLLM exposes Prometheus metrics and logging for these.

Step 5 — Isolate one change. Re-run with an unchanged model and hardware while varying only the runtime; or fix runtime and compare GPU families. Where feasible bisect configuration changes or versions. Avoid toggling compiler, kernel and batching settings simultaneously.

Step 6 — Profile a small sample. Use Nsight Systems for GPU timelines/kernel gaps and PyTorch Profiler for richer operator shapes/stacks/memory diagnostics where needed. vLLM cautions that profiling can significantly slow inference—measure final performance in separate unprofiled runs.

Step 7 — Validate the fix and prevent recurrence. Repeat matched load sweeps and quality tests, compare p95/p99 TTFT/ITL and goodput, canary under approved traffic, and add the failing workload to CI regression gates.

5. Common regression signatures and root causes

• TTFT rises, queue wait grows: new scheduler token budgets, reduced ready replicas, slower routing, uneven long prompts, or more preemption reducing service capacity.
• TTFT rises with stable queue: tokenization/prefill kernel regression, compilation fallback, changed prompt length, lower prefix hit rate or more costly KV transfer.
• ITL rises, TTFT mostly stable: slower attention/decode kernels, more active decode sequences, GPU HBM bandwidth contention, CPU orchestration gaps or NCCL collectives.
• GPU memory rises after model change: a larger weight set, changed KV-head count/cache dtype, compilation workspace, less efficient allocation or changed max_model_len.
• Prefix-cache hit rate falls: changed prompt prefix, tokenizer/template, cache hash/eviction settings, router affinity or model compatibility boundaries.
• Preemptions surge: KV cache too small for concurrency, changed batch-size limits or attention layout. In vLLM V1, common preemption uses RECOMPUTE, which increases repeated GPU work.
• Throughput drops after moving to a newer GPU: mismatched CUDA/kernel builds, lower actual clocks, PCIe/NVLink bottlenecks, parallelism/placement mismatch or an unsupported optimized path. A faster GPU specification does not guarantee a faster *whole system*.
• Startup becomes slower but steady-state inference is normal: compilation cache miss, extra CUDA graph capture, model weights staging or changed optimization mode. Distinguish cold-start latency from ongoing TTFT/ITL.

6. A controlled one-factor experiment

Suppose p95 ITL worsens after a runtime-and-driver bundle update. Test safely:

1. Old runtime + old driver (known baseline).
2. New runtime + old driver only if that combination is supported.
3. Old runtime + new driver only if supported.
4. New runtime + new driver (candidate).

If a compatibility matrix forbids one combination, do not run unsupported mixes. Use supported intermediate releases or separate matched environments instead. The objective is to attribute the regression to an independent factor or an interaction, not to force an impossible experiment.

Keep model, hardware, power settings, exact traffic replay and percentile methodology fixed. Re-run with different seeds and realistic workloads.

7. Production incident: new runtime disables an optimized path

Incident: A model-serving upgrade initially looks healthy in basic readiness checks, but p95 TTFT rises from 0.8 s to 1.2 s and goodput drops from 12 to 9 requests/s after production traffic ramps up.

Root-cause analysis:

• Trace comparison reveals queue wait rising on heavily loaded replicas.
• Scheduler metrics show KV preemptions and recomputation became common for long-context requests.
• The new runtime's cache allocation or concurrency configuration leaves fewer effective KV blocks despite unchanged GPU count.
• The bottleneck is therefore not necessarily the model weights themselves but changed engine configuration and available KV headroom.

Engineering fix: Restore a version-compatible cache/concurrency policy; test reasonable adjustments to KV memory allocation and batch-token/active-sequence limits; verify no OOM; preserve intended prefix-cache behavior; retest under realistic mixed input lengths. Apply canary rollout/rollback gates.

Prevention: Pin a deployment manifest with full model/runtime/hardware/flags, alert on unexpected preemptions, and require separate cold/warm benchmark plus long-context p95/goodput regression tests before promotion.

This is a hypothetical incident; the investigation chain is illustrative, not an assertion that a specific vLLM release has this defect.

8. Monitoring and release gates

• Version labeling: model hash, runtime build, GPU type, attention/quantization backend and rollout cohort for all serving/tracing metrics.
• Performance: p50/p95/p99 TTFT, ITL, queue wait, output token counts, SLO-conforming goodput and actual offered rate.
• Resource use: vllm:num_requests_running, vllm:kv_cache_usage_perc, vllm:prefix_cache_hits, vllm:prefix_cache_queries, prompt/generation tokens and preemptions where exported. Metric names and availability vary by installed version.
• Correctness: schema validity, task pass rate, model regression suites and safety-policy checks.
• Reliability: streaming disconnects, 429/503/5xx, latency-based admission rejections, cancellation cleanup and worker crashes.
• Cost: GPU-hours per successfully completed SLO-conforming request.
• Noise control: enforce a minimum sample count, stratify by prompt/output length, and use consistent warmup/observation windows. One surprising sample is not proof of a statistically persistent p99 regression.

Interview-ready answer: “I would reproduce the slowdown under the same model, traffic, GPU and cache conditions, separate queueing from prefill and decode spans, then correlate the affected stage with KV/preemption, kernel, communication and scheduler telemetry. I would isolate changes with supported old/new combinations and use short Nsight/PyTorch traces for cause—but benchmark without profiling to avoid observer overhead. I'd roll back unsafe candidates, verify both latency and answer quality, and add the scenario to automated release gates.”

Research references:

• vLLM — Optimization and Tuning
• vLLM — Profiling vLLM
• vLLM — Metrics
• vLLM — Benchmarking CLI
• NVIDIA — Nsight Systems