How do you design production ML inference observability, SLOs, and incident response?

Production ML inference observability links user-visible reliability to the exact model release, input features, dependencies and decisions that produced each prediction. Define service-level indicators (SLIs) and objectives (SLOs) for successful, on-time, valid responses; use metrics, traces and privacy-safe structured logs to locate failures; page on actionable business impact; and audit recovery against the same objectives.

Prerequisite boundary: Question 65 covered training-pipeline run histories. Questions 119 and 126–129 covered serving modes, autoscaling, admission control, fallback, high availability and disaster recovery, including their individual metrics. Question 130 unifies production inference incident detection, SLO mathematics, burn-rate alerts, evidence correlation and on-call response, without teaching drift-detector statistics or repeating the architecture and recovery mechanisms themselves.

Practical production incident

A delivery company serves late-risk/r28 to 200 checkout requests per second. Its dashboards show 99.9% HTTP 200 responses, yet some customer checkouts are 600 ms late, and another group receives predictions from stale traffic features. The gateway reports the API as healthy, but customers are not getting useful decisions on time.

The response requires distinct signals:

• Transport success: Did the API return at all, without a server error?
• Timeliness: Did it return before the caller's actual deadline?
• Prediction validity: Was the schema correct and the result from the intended model/policy and valid feature contract?
• Data freshness: Were required online features current enough for this decision?
• Mode provenance: Did the response come from the primary, an explicitly approved fallback, or a cache?
• Business effect: Did the warning/decision rate or downstream outcome change for a relevant user cohort?

A 200 from a fallback is not automatically “primary inference recovered.” An unready model or broken dependency can still create invalid customer outcomes despite high infrastructure uptime.

Step 1: define user-centered SLIs and SLOs

For an online predictor, define a good request: eligible requests that produce a policy-approved, correctly attributed prediction under the end-to-end deadline and freshness contract. The denominator is all *eligible user requests* at the edge, including overload rejections and timeouts, not just requests admitted by model workers.

Example SLOs (illustrative, not universal):

• On-time useful prediction SLO: ≥ 99.5% of eligible checkout requests complete successfully within 250 ms over a rolling 30-day window.
• Schema validity SLO: ≥ 99.99% of served predictions match the contracted version and output constraints.
• Freshness guardrail: 0 use of forbidden stale traffic features for designated high-impact decisions (subject to reliable detection).
• Latency diagnostic: p95 ≤ 250 ms measured at a defined boundary; percentile alone is not a substitute for the good-event SLO.
• Fallback guardrail: Fallback share within an explicitly approved incident policy and limited duration; report it separately.
• Batch mode: Completed, reconciled prediction data available before the consumer deadline with no partial publication.
• Streaming mode: Complete valid outputs within event-to-output freshness bounds, accounting for watermark and replay lag.

If a synchronous service has a 250 ms deadline, the critical SLI can be expressed as good eligible requests / all eligible requests. Separately track hard server failures, client cancellations, invalid inputs, stale-feature outcomes and fallback use. Define client-caused invalid requests and exclusion policies *before* calculating the denominator; never redefine it during an incident to make the metric look better.

Numerical example 1: error budget

Suppose 30 days contain 10,000,000 eligible requests and the on-time useful prediction target is 99.5%.

• Permitted bad events = 10,000,000 × (1 − 0.995) = 50,000.
• If 25,000 time out, 10,000 are rejected under overload, and 5,000 return an invalid or too-stale prediction, with mutually exclusive classifications, total bad events = 40,000.
• Remaining budget = 50,000 − 40,000 = 10,000 bad events, or 20% of the original budget.
• If another 15,000 requests violate the contract, the budget is exceeded by 5,000 events. This is a policy signal to pause nonessential risk-increasing releases and prioritize reliability.

The groups must be disjoint for simple addition: a timeout with a stale feature should be counted as one bad user request in the top-level SLI, while diagnostic reason labels may overlap. Error budgets based on request proportions do not directly measure duration of regional outage; volume and affected customers matter.

Numerical example 2: budget burn and paging

Assume the rolling reliability SLO is 99.5%: allowed bad fraction 0.005. In the previous 5 minutes, 10,000 requests were eligible and 300 were bad.

• Observed bad fraction = 300 / 10,000 = 0.03 = 3%.
• Burn rate relative to allowed fraction = 0.03 / 0.005 = 6×.
• At a steady 6× burn, the budget is spent six times as quickly as the long-run allowance, but a single 5-minute observation alone is not proof that 30-day performance will fail.
• Alert policy: use a fast and a longer corroborating window for urgent pages, plus a slower burn-rate ticket. Demand sufficient request volume and compare with current regional/cohort impact; suppress duplicate pages from a known incident.

Google SRE recommends user-centered SLOs, an explicit reliability error budget and actionable pages rather than alerting on every infrastructure fluctuation. An exact burn-rate threshold requires an agreed paging strategy; the 6× calculation is explanatory, not a mandatory universal rule.

Numerical example 3: tail latency is not average latency

Suppose 1,000 checkout inference requests have 950 responses at 100 ms and 50 at 800 ms.

• Mean latency = (950 × 100 + 50 × 800) / 1000 = 135 ms.
• Yet the slowest 5% are 800 ms, which violates a 250 ms deadline.
• Using a conventional nearest-rank definition, the observed p95 falls at the 950th value (100 ms), while p95 may differ under other interpolation/window definitions. A p95 reported as 100 ms can therefore hide the slowest 5%.
• The directly calculated good-event fraction is 950/1000 = 95%, revealing 50 late user responses even though the mean looks excellent and the p95 boundary can just pass.

Use well-defined histogram buckets/percentile methods, and prefer the fraction completing under the actual deadline as the user-centered SLO. Prometheus explains why per-replica quantiles are not generally aggregatable and why histograms enable aggregate latency analysis.

Step 2: instrument a complete prediction trace

Give each eligible request a trace or correlation ID that survives ingress, preprocessing, online feature retrieval, model execution, postprocessing and any approved fallback. A privacy-safe structured diagnostic record might contain:

• Timestamp, trace ID, operation name, service, region/zone, tenant tier (low cardinality) and response class.
• Immutable serving release, model digest/version, feature-contract revision and explicit primary/fallback/cache path.
• Request lifecycle: accepted/rejected, queue wait, feature-fetch duration, model execution duration, serialization, overall latency and deadline remaining.
• Number of required features missing, freshness violation boolean and a *bucketed* feature age; do not log sensitive raw feature values by default.
• Request outcome: valid/on-time or precise bad-event classification; whether a downstream business action was issued.
• For batch: run ID, partition/manifest ID, completion time and eligible-row reconciliation.
• For stream: event identity, source offset/checkpoint, event time, processing time, lag and sink deduplication outcome.

Use OpenTelemetry concepts and appropriate semantic conventions for HTTP and inference spans. OpenTelemetry GenAI conventions are specialized and some definitions are evolving; a conventional classifier should not blindly emit token-oriented GenAI metrics. Version your custom ML inference fields and instrument consistently across stable, canary and fallback releases.

Data minimization: Never place raw customer inputs, personal identifiers, auth headers, secrets or unrestricted prediction payloads into high-volume telemetry by default. Use opaque IDs, sampling, role-based access, retention controls and separately governed debugging captures.

Step 3: dashboards and diagnostic flow

Create a small service health dashboard plus drill-down views instead of one enormous unsearchable graph.

Top-level operational SLO view

• Eligible request rate, good/bad SLI, remaining error budget, burn rate, latency distribution, overload rejections and timeouts.
• Status by live release and region; primary/fallback decision mix and critical feature freshness.

Dependency investigation view

• Feature-store p95/p99, missing/stale feature counts, cache hit rate, circuit-breaker state, upstream 429/503 and model-batch/queue times.
• Ready/desired replicas, accelerator memory, model warmup time, concurrency, autoscaler lag and routing weights.

ML correctness view

• Inference schema/contract failures, scores outside valid range, missing-value policy use, prediction distribution shifts by model version, label maturity and validated outcome metrics.
• Separate unlabeled feature/prediction drift *signals* from delayed ground-truth accuracy. A changing prediction histogram may reflect a real shift in request mix; do not automatically diagnose “model degradation” from one chart.

Mode-specific view

• Batch: due-time slack, complete/expected partitions, committed output manifest, failure retries and last successful consumer-visible publication.
• Streaming: source arrival and processing rate, consumer lag, event-to-output latency, watermark delay, checkpoint age, replay duplicates and sink errors.

Step 4: incident triage runbook

Incident: good-prediction SLO suddenly falls after a deployment

1. Confirm user impact: Does the bad-event rise reflect actual user requests or a broken instrument? Check client-side or external synthetic predictions and denominator stability.
2. Segment quickly: By region, stable/candidate release, major user cohort, feature schema and primary/fallback mode. Avoid personal identifiers in high-cardinality metric labels.
3. Decompose latency and errors: Find whether queue wait, feature fetch, model computation or serialization grew. Check rejections, retry amplification, stale features and bad payloads.
4. Identify recent changes: Model/image/feature-contract rollouts, router weights, upstream schema/materialization, node/capacity changes, dependency or region incidents.
5. Protect users: Abort a failing canary, reduce load, isolate a dependency, invoke only approved fallback, or trigger HA/DR runbooks as appropriate. These actions reuse Questions 122 and 126–129; this lesson explains *how evidence chooses the action*.
6. Verify recovery: Observe externally successful valid predictions and normal error-budget burn, not just green Pod readiness. Check that known affected cohorts and feature freshness recovered.
7. Preserve evidence: Incident timeline, traces, release digests, route changes, request classification and outcome maturity. Assign an owner for delayed model-quality review.
8. Post-incident action: Make one concrete prevention change, test the alert against replayed metrics and verify that the next rollout includes the regression guardrail.

Incident: infrastructure healthy but predictions became suspicious

• Check whether scores exceed agreed bounds, missing-feature defaults spiked, an upstream unit changed, or the model/feature versions are mismatched. Compare like-for-like cohort traffic and inspect delayed labels. Do not assume throughput alerts can detect semantic prediction errors.

Incident: alert flood but no user impact

• Inspect paging rules and dashboard labels. If a single known feature-store outage fires dozens of derivative alerts, group under one actionable incident while keeping diagnostic signals visible.

SLO-driven alert policy and incident ownership

• Page now for severe customer harm, sustained fast budget burn, critical invalid decisions, broad failed inference or dangerously stale required features.
• Ticket for follow-up for slow error-budget burn, elevated fallback use within approved limits, anomalous cost or low-impact contract errors.
• Record without paging for ordinary model score variation without demonstrated quality harm, routine autoscaling or expected batch warmup.

Specify an on-call runbook with service owner, feature-store owner, platform owner, model steward, escalation contacts, safe shutdown actions, known-good release and post-incident reviewer. One alert should describe the user symptom, current magnitude, affected cohorts, likely dependencies and a direct runbook location rather than sending a CPU graph without context.

Practical conceptual SLO rule (not vendor-specific production config)

YAML
service: delivery-risk-inference
window: rolling_30_days
sli:
  denominator: eligible_edge_requests
  numerator: eligible_responses_with_valid_approved_prediction_under_250ms
objective: 0.995
diagnostics:
  - status_and_overload_reason
  - request_latency_histogram
  - model_release_and_fallback_mode
  - feature_freshness_and_contract_errors
  - dependency_latency_and_inference_duration
  - region_and_approved_cohort
alerts:
  fast_burn: page_after_sustained_multiwindow_violation
  slow_burn: ticket
  critical_invalid_decision: page
  telemetry_missing: separate_instrumentation_alert
runbook: operations/inference-slo-incident

This YAML is a design illustration, not an executable Prometheus/Alertmanager configuration. In practice, define counters, histogram boundaries and scrape coverage, test aggregation and label-cardinality policy, and explicitly choose windows/minimum volumes to avoid alerts driven by tiny samples.

Production failure scenarios, trade-offs and common mistakes

• HTTP success hides stale or invalid predictions: Keep quality/validity requirements in the good-event definition and trace model-feature provenance.
• Accepted-only SLO hides rejected checkout requests: Measure from the user-facing edge, including 429/503, timeouts and approved fallback semantics.
• Average latency seems healthy: Measure tail latency and on-time fraction; use aggregatable histograms rather than averaging per-Pod p95 values.
• Metric cardinality spikes: Do not put request IDs, customer IDs or raw model input values in time-series labels. Put opaque IDs in sampled traces/logs with access controls.
• Labels are delayed: Operational SLO is immediate; causal business and prediction-accuracy analysis must wait for sufficiently mature labels.
• Monitoring backend fails: Alert on telemetry coverage separately. An absent metric is not evidence of zero failures.
• Sampling hides rare harmful outcomes: Use unsampled critical decision counters and carefully governed rare-event reporting; sampling may be suitable for ordinary traces.
• Too many pages: Require an actionable symptom and multiple windows where appropriate; keep warnings and diagnostics out of urgent paging.
• Auto-rollback from weak data: A single statistically noisy prediction-distribution metric should not automatically reverse a safe production release. Define severe invariants and guardrails explicitly.
• Only model metrics, no dependencies: Break a trace into feature, queue, model and output spans; ML failures commonly originate outside model weights.
• Only infrastructure metrics, no model behavior: A healthy cluster may still return stale features or incorrectly interpreted scores.
• Business cost ignored: Include requests per useful on-time prediction, additional fallback costs, log sampling budgets and operational toil alongside error budgets.

Practical operator checklist

• Versioned user-centered good-event definition and policy-approved exclusions.
• Error budget with clear burn-rate alert ownership and chosen measurement windows.
• Per-release, per-region metrics across primary, fallback, rejected and invalid decisions.
• End-to-end traces connecting feature requests, inference execution and user-visible response.
• Structured privacy-safe logs and tested metric-cardinality/retention limits.
• Synthetic prediction probes and alarms for telemetry gaps.
• Runtime, data validity and delayed-label quality dashboards with separate confidence semantics.
• Actionable incident runbook that integrates rollout, overload, fallback and DR paths.
• Recovery verification and postmortem evidence, including concrete regression tests.

Takeaway: A dependable ML service is not just “up”; it delivers a valid, sufficiently fresh prediction within the customer's deadline. Measure that outcome directly, correlate it to versions and dependencies, use error budgets to prioritize reliability work, and make incident response accountable and testable.