How do you build production observability dashboards and alerts for LLM inference latency, throughput, KV-cache pressure, and errors?

Production LLM inference observability joins user-facing service-level indicators (SLIs), engine metrics, GPU telemetry, and request traces into dashboards and alerts that explain what is failing, for whom, where, and why. A useful dashboard distinguishes demand, accepted work, successful/SLO-compliant completions, queueing, prefill, decode, and KV-cache pressure. Alert on customer impact and leading saturation signals, not merely high GPU utilization.

This lesson extends Question 189 (profiling fundamentals), Question 257 (goodput calculation), and Question 266 (regression investigation) by focusing on a production observability architecture, PromQL, alert design, runbooks and safe telemetry cardinality.

1. What a production dashboard should answer

• Is the customer experience healthy? Show p50/p95/p99 time to first token (TTFT), inter-token latency (ITL) or per-request time per output token (TPOT), total latency, and the fraction of original user requests meeting the agreed SLOs.
• Is demand higher than safe capacity? Plot offered request rate, accepted rate, completed request rate, cancellations/rejections, waiting requests, running sequences and queued prompt tokens.
• Where is the time going? Split queue wait, prefill, optional KV transfer, decode and response/network overhead using request traces and histograms.
• Is the model memory-constrained? Show fraction of used KV blocks, cache evictions/preemptions, prefix-cache hit fraction, GPU HBM and active-context length distributions.
• Are errors increasing? Display 429/503, timeout, inference 5xx, worker restart and stream-abort counts separately; classify expected user cancellations rather than treating all as server defects.
• Is the service efficient? Include prompt/output tokens per second, SLO-compliant goodput, cost per successful/SLO-conforming request, and availability.

2. Metrics that vLLM exposes

Current vLLM v1 Prometheus-compatible metrics include, subject to the installed version and feature flags:

• Running and queued requests: \vllm:num_requests_running\ and \vllm:num_requests_waiting\.
• KV-cache pressure: \vllm:kv_cache_usage_perc\ is a 0–1 fraction of used KV blocks (multiply by 100 for percent).
• Throughput counters: \vllm:prompt_tokens_total\, \vllm:generation_tokens_total\ and \vllm:request_success_total\ (finish-reason labels matter).
• Latency histograms: \vllm:time_to_first_token_seconds\, \vllm:inter_token_latency_seconds\, \vllm:request_time_per_output_token_seconds\, \vllm:e2e_request_latency_seconds\, and request queue/prefill/decode time histograms.
• Prefix-cache counters: \vllm:prefix_cache_hits\ and \vllm:prefix_cache_queries\; calculate a ratio over the same time window.
• Admission rejections: \vllm:admission_rejections_total\ where supported.
• GPU metrics: use NVIDIA DCGM exporter or equivalent monitoring for GPU memory, compute, power, temperature and other device signals.

Check the actual /metrics payload and engine version before deploying a copied dashboard. Old vLLM versions have different metric names and do not expose every newer metric. For example, legacy GPU-cache utilization metrics differ from the current \kv_cache_usage_perc\.

Critical distinction: ITL records per-stream-output gaps; request-level TPOT estimates average decoding time per output token per request. They differ when output events contain multiple tokens. Do not label ITL p95 as TPOT p95 interchangeably.

3. Beginner-friendly numerical example

A fictional five-minute window has:

• 3,000 original requests, of which 2,850 succeed, 90 fail, and 60 are rejected or time out.
• 2,400 of those original requests finish successfully and meet the joint TTFT and TPOT objectives.
• A configured target of 99% successful SLO-compliant service.

Compute:

1. Overall success fraction = \2,850 / 3,000 = 95%\.
2. Joint SLO attainment = \2,400 / 3,000 = 80%\.
3. SLO goodput = \2,400 / 300 = 8 requests/s\.
4. Fraction violating the success-plus-latency objective = \(3,000 - 2,400)/3,000 = 20%\.
5. If the permitted failure budget is 1%, the short-window error-budget burn multiple is \20% / 1% = 20×\.

Interpretation: A 20× burn means this *window's* error-budget consumption is twenty times the long-term sustainable rate, not that 20% of an entire monthly budget has already been exhausted. A paging policy should require measured volume and appropriate multi-window burn-rate criteria rather than alerting on one tiny interval.

Now suppose \vllm:kv_cache_usage_perc\ rises from 0.65 to 0.96 while waiting requests climb from 3 to 70 and p95 TTFT rises from 0.7 s to 2.5 s. This combination suggests an urgent saturation investigation. High KV usage alone is not proof of a failure: examine preemptions, evictions, active lengths and SLO impact.

All numbers are illustrative.

4. A practical metric pipeline

1. Instrument gateway and model engine. Export request outcome counters and histograms from a trusted gateway, and serving/KV metrics from each engine replica.
2. Scrape Prometheus-compatible endpoints using ServiceMonitor/PodMonitor or a managed equivalent; verify that labels identify model revision, deployment, region and instance.
3. Create Grafana dashboards with fleet-wide customer-impact panels at the top and drill-down panels for queue, prefill, decode, KV memory and GPU devices.
4. Correlate with OpenTelemetry traces, propagating an opaque request ID across gateway, router, prefill, optional KV transfer, decode and stream output.
5. Use logs for diagnosis, with structured reason codes and version fingerprints, not for storage of raw sensitive prompts by default.
6. Attach runbooks to actionable alerts, including on-call severity, likely causes, dashboards, safe mitigations, and escalation.

Protect metrics/traces from unnecessary tenant IDs, raw request IDs, prompt contents, bearer tokens and other high-cardinality/sensitive labels. Use authorized tracing storage with privacy-reviewed retention.

5. Practical PromQL examples

The following queries illustrate common vLLM v1 metrics. Confirm names and histogram buckets in the installed version; ensure all required dimensions are consistent.

P95 fleet TTFT over five minutes: \\\promql histogram_quantile( 0.95, sum by (le) ( rate(vllm:time_to_first_token_seconds_bucket[5m]) ) ) \\\

P95 TTFT by model name: \\\promql histogram_quantile( 0.95, sum by (model_name, le) ( rate(vllm:time_to_first_token_seconds_bucket[5m]) ) ) \\\

Output tokens per second: \\\promql sum(rate(vllm:generation_tokens_total[5m])) \\\

Current queue size and KV occupancy: \\\promql sum(vllm:num_requests_waiting) max by (model_name) (vllm:kv_cache_usage_perc) \\\

Windowed prefix-hit fraction: \\\promql sum(rate(vllm:prefix_cache_hits[5m])) / clamp_min(sum(rate(vllm:prefix_cache_queries[5m])), 1e-9) \\\

Important: calculate rates from individual counters before aggregation to handle resets. For p95, aggregate histogram buckets before \histogram_quantile\; never average pod-level p95 values. A Prometheus histogram percentile is an approximation based on its buckets. Define buckets around actual SLO boundaries; too many buckets increase cardinality and storage costs.

For business-level joint TTFT+TPOT goodput, emit an independent end-of-request count from an application/gateway evaluator that knows both observations and request success status. You cannot derive a correct joint pass count merely by combining two independent histograms.

6. Alert design: page versus investigate

Page on real or imminent customer impact:

• Multi-window, multi-burn-rate SLO violations at a sufficient sample volume.
• Sustained high p95/p99 TTFT or TPOT with concurrent increases in queue delay, timeouts or failed requests.
• Sudden request failure/rejection spike, unavailable serving replicas, or a confirmed severe security/quality issue.

Create warning/ticket alerts for leading signals:

• Sustained queued work and rising oldest-request age while offered demand exceeds successful completions.
• KV occupancy nearing the benchmarked unsafe region plus preemption/eviction or TTFT impact.
• Falling prefix hit fraction after a new template/router release.
• Worsening GPU memory fragmentation, model restart rate, GPU hardware errors, cost per good request or startup latency.

Avoid noisy alerts:

• A single high GPU-utilization value may be healthy batching.
• A 99% KV-cache gauge may be normal under some workloads; calibrate thresholds against actual preemptions/SLOs.
• p99 latency at low volume is unstable; enforce a minimum request-count threshold and sensible windows.
• Separate alerts per model/workload/region but avoid one page per replica for one fleet-wide incident.
• Track missing metrics and scrape failures explicitly; missing data is not the same as a healthy zero.

7. Production incident: KV pressure hidden behind aggregate averages

Incident: A chat endpoint's fleet-average TTFT looks stable, but users with 16k–32k-token prompts experience p99 TTFT of 7 seconds, frequent timeouts and poor streaming quality.

Debugging steps:

1. Slice TTFT by prompt-length buckets, engine revision and replica. Investigate queue time versus prefill time.
2. Observe one replica at 95% KV block use, repeatedly preempting/recomputing long contexts, while others remain at 60%.
3. Inspect prefix reuse, router placement, max model length and per-replica batching/active-sequence settings.
4. Compare admission rejections and completed-request goodput by tenant/workload to see whether errors were masked by success-only latency charts.

Root cause: The dashboard displayed only fleet average latency and GPU utilization. It hid request-length distribution, unbalanced routing, KV/cache pressure and failure exclusions.

Engineering fix: Build length-sliced p95/p99 and joint goodput panels, add queue/KV/preemption correlations, rebalance eligible work using capacity-aware routing, and tune admission/backpressure. Keep paging tied to user impact, not a lone cache gauge.

Prevention: Test dashboard/alert coverage using mixed long-prompt load, replica failure and forced KV pressure. Require end-to-end outcome instrumentation as a production release gate.

8. Dashboard structure and runbook

A useful first-page service health dashboard contains:

1. Request arrivals/accepted/completed/rejected/failed, with error codes.
2. Joint SLO success and error-budget burn with p95/p99 TTFT and request-level TPOT.
3. Per-model/region routing distribution, waiting requests, oldest queue age and active sequences.
4. Prompt/output tokens per second and average/percentile token lengths.
5. KV occupancy, prefix-hit fraction, evictions/preemptions and GPU device memory.
6. Canary/stable version comparison and timestamps for deploys, autoscaling events or hardware changes.

Example alert runbook: If TTFT suddenly rises, first check waiting queue and admission rate. If queue wait rises with stable prefill, look at capacity, routing, retries and KV preemption. If queue is stable but prefill grows, inspect uncached prompt length, prefix reuse, model/runtime/kernel changes, and GPU communication. If TTFT is stable but TPOT worsens, focus on decode batching, KV bandwidth, GPU contention and collectives. Roll back unsafe releases or shed excess work as the agreed SLO policy permits.

9. Interview-ready answer

“I'd expose vLLM engine histograms/counters and gateway outcomes through Prometheus, build Grafana panels for p95/p99 TTFT, request-level TPOT, queue depth, KV occupancy, token throughput and successful SLO-goodput, and join request traces across prefill/decode. I'd use multi-window burn-rate paging for customer impact, separate leading saturation warnings, and keep labels low-cardinality. For debugging, I'd slice by model, prompt length and release version and use a runbook that ties queueing, KV preemption and kernel changes to user-visible latency.”

References:

• vLLM — Production Metrics
• vLLM — Design of Metrics and Grafana
• Google SRE Workbook — Alerting on SLOs
• Prometheus — Histograms and Summaries
• NVIDIA DCGM Exporter