How do you run safe chaos engineering and game-day experiments for production ML inference services?

Chaos engineering is hypothesis-driven fault injection that tests whether an ML inference service continues making safe, timely decisions—or recovers within a defined objective—when real dependencies fail. A game day extends the exercise to the people, alerts, runbooks, approvals, customer communications and evidence needed to manage an incident. The goal is to discover resilience gaps without causing avoidable user harm.

Curriculum position: Question 121 validated preproduction serving behavior, Question 126 addressed capacity, Question 127 bounded overload, Question 128 defined fallback safety, Question 129 designed HA/DR, and Question 130 instrumented SLOs and incident response. Question 131 tests those mechanisms with controlled, reproducible failure experiments; it does not re-explain the failover architecture, retry mechanism or error-budget formulas. Question 65's training-pipeline observability is separate from live prediction failure experiments.

Beginner-friendly incident

A delivery company operates a production checkout risk predictor (late-risk/r28). Its dashboard reports healthy Kubernetes Pods, but the company has never tested what happens when an online traffic feature becomes unavailable. During an unplanned outage, 30% of checkouts time out and the fallback sometimes returns an unsafe low-risk score.

Instead of waiting for the next incident, the team proposes a bounded hypothesis:

> If traffic-feature reads fail for one test tenant for three minutes, authorized low-impact requests should use the approved minimal-feature model when eligible, high-impact decisions should fail closed, and unaffected tenants should maintain their normal latency.

A controlled experiment turns that expectation into observed evidence. It must not inject faults into payments, real customer notifications, high-impact decisions or unrelated tenants.

Core vocabulary and engineering rules

• Steady state: Baseline user-facing SLI, valid-prediction fraction, end-to-end latency, feature freshness, primary/fallback mix and error budget burn before injection.
• Hypothesis: A falsifiable statement about the observable response to one specific fault.
• Fault: An intentional, bounded perturbation such as network delay, feature-store 503, model process restart, invalid feature schema, node loss or artificial request burst.
• Blast radius: Maximum affected tenants, regions, traffic share, Pods, dependencies, request rate and experiment duration.
• Abort/stop condition: A predeclared threshold that halts injection before unacceptable harm; always include a human-accessible kill switch.
• Recovery test: Evidence that the service truly returns to its pre-experiment normal operating mode, not just that fault injection ended.
• Game day: An orchestrated practice incident testing telemetry, technical responders, business decision-makers and coordination.

Never run arbitrary destructive faults in real production merely because chaos engineering is fashionable. Begin with a disposable test environment, then a production-like staging environment; move to a limited production slice only with approval, robust isolation and tested rollback.

Seven-step safe experiment design

1. Inventory the prediction path. Map ingress, authorization, queue, model weights, tokenizer/preprocessor, feature retrieval, cache, accelerator, inference worker, business rules, output sink, audit logger and operational telemetry. Name single points of failure and shared dependencies.
2. Rank realistic failures. Prioritize failure modes based on observed incidents, dependency criticality and severity: feature-store slowdown, stale feature updates, GPU memory failure, throttling, rolling image pull, broken model schema, instance or zone outage, and metric/alert gaps. Do not prioritize theatrical faults over known vulnerabilities.
3. Define one hypothesis and an observable oracle. Example: "At most 0.5% of eligible test requests violate the deadline when one isolated replica is terminated, and all served scores retain their expected model digest and freshness." Record the exact population and metric windows.
4. Require safety prechecks. Validate last-known-good release and feature compatibility, abort permissions, emergency capacity, stop-condition alarms, audit logging, incident owner, time window, and absence of an active high-severity outage.
5. Start with minimal scope. Use synthetic traffic or a preauthorized internal tenant first; restrict target Pods with labels and namespaces, enforce a hard duration and respect GPU/feature-service quotas. Escalate one variable at a time.
6. Observe during fault and restoration. Measure user-centered SLO, per-release prediction validity, actual fallback actions, retries, deadline breaches, dependency pressure and recovery latency. Verify downstream business side effects and control-plane state.
7. Capture the outcome and create a regression. Keep the hypothesis, target IDs, owner, authorization, baseline, injection logs, stop triggers, results, counterfactual limits, and remediation ticket. Repeat after corrections; automate only those scenarios whose isolation and invariants are proven.

Counterfactual caution: If traffic, deployment and dependency configurations change during the same window, the injected fault may not be the only cause of observed degradation. Use matched baselines, hold important changes constant and interpret results cautiously.

Numerical example 1: a pod-loss hypothesis

An endpoint serves 180 requests/second and currently runs nine warm replicas distributed across three zones. From the established capacity study each replica sustains 30 RPS at the targeted SLO after headroom.

• Normal planned capacity is 9 × 30 = 270 RPS, so measured demand uses 66.7% of planning capacity.
• Inject failure of one *specifically tagged, noncritical* replica, leaving eight replicas and 8 × 30 = 240 RPS of nominal planned throughput.
• Remaining nominal capacity exceeds 180 RPS by 60 RPS, under the simplifying assumptions of even load and healthy dependencies.
• Hypothesis: The router removes the unavailable target promptly, response validity stays at baseline, and 99.5% or more of eligible experiment requests are on time.
• Measure: Time from fault to target removal; requests that reached the failed Pod; ready and serving capacity; p95/p99; 429/503; and valid predictions by tenant.

Failure discovery: The remaining eight replicas may have enough aggregate theoretical throughput but the ingress might continue sending requests to the dead replica for 20 seconds. With a uniform nine-way distribution, 180/9 × 20 = 400 requests could hit that target in that window if no other routing protection intervenes. This illustrates why capacity arithmetic does not prove fast failure detection.

Do not actually terminate Pods in production unless targets, blast radius and stop conditions are controlled and approved.

Numerical example 2: error budget and experiment abort

A test tenant receives 40 eligible RPS. In a 5-minute fault window, it generates 40 × 300 = 12,000 requests. For this scoped experiment the agreed maximum bad-request fraction is 0.5%.

• Maximum bad-event budget for that test window = 12,000 × .005 = 60 requests, assuming an exact window and this particular acceptance rule.
• At minute two, 80 requests have already violated the safe-response contract.
• Even if later requests are perfect, the experiment has already exceeded the 60-event full-window allowance.
• Decision: Trigger the abort policy, stop the fault, restore the path, verify correct model responses and page the responsible operator if thresholds warrant it.

Important: A full-window allowance should not be the *only* live abort signal. Use a much faster hard stop for critical invalid decisions, any unauthorized side effect, and sustained short-window SLI/latency deterioration. A fault that causes one dangerous action can be unacceptable even when numerical availability remains “green.” The overall service's rolling error budget from Question 130 must be considered separately from the experiment's own exposure cap.

Numerical example 3: feature-service brownout

A model requires a remote feature service. Under normal operation feature fetch takes 30 ms, model execution 50 ms, and other processing 20 ms, totaling 100 ms before queuing.

Inject a controlled +100 ms delay *only for the isolated test cohort*:

• Expected processing duration becomes approximately 30 + 100 + 50 + 20 = 200 ms if stages are sequential.
• If queueing adds 70 ms at peak, end-to-end latency becomes approximately 270 ms.
• At a 250 ms request deadline, this request misses by 20 ms, even though feature service and model both returned successfully.
• A fallback that starts only after a 200 ms feature timeout and then needs 80 ms cannot fit the remaining 50 ms budget; the hypothesis must consider the entire deadline.
• Outcome: Fail the degraded-path latency hypothesis, lower the dependency timeout or revise eligibility/routing, then re-test under the same representative load.

The numbers are illustrative: real latency stages may overlap and tails are not sums of averages. Use measured traces, percentiles and explicit request deadlines.

Failure injection matrix: what to test and what to observe

Feature dependency latency or 503

• Inject: bounded timeout, 429/503 or stale response to a test identity.
• Expected: timeout/circuit breaker works, backup is independently valid or service fails clearly, no retry storm; critical feature freshness remains enforced.
• Watch: feature spans, remaining deadlines, fallback count, 5xx, score provenance and upstream call amplification.

Model-process crash or OOM

• Inject: restart a disposable canary replica or constrained container in staging.
• Expected: ingress stops routing to an unready target, replacement loads exact signed model/image digests, valid response service resumes.
• Watch: availability, restart loops, model download, warmup and GPU memory.

Invalid preprocessing/feature contract

• Inject: schema mismatch in a synthetic request or an isolated mock feature endpoint, never poison real training/serving data.
• Expected: invalid inputs are rejected safely; no silently coerced units or model outputs.
• Watch: contract violations, missing fields, model/feature version mismatch and customer decision consequences.

Capacity burst and queuing

• Inject: capped artificial arrivals into isolated capacity.
• Expected: concurrency and queue limits enforce deadlines; requests reject clearly rather than time out deep in inference.
• Watch: queue age, admitted/rejected, retries, GPU utilization and P99.

Zone or regional impairment

• Rehearse in staging or approved narrowly scoped failover exercise. Validate warm capacity and data freshness in surviving locations.
• Watch: real route distribution, recovered valid predictions, RTO/RPO where relevant, duplicate decisions and cross-region lag.
• This tests Question 129's *assertions*; it does not repeat how to architect failover.

Telemetry outage

• Stop only synthetic trace/metric export for a test service.
• Expected: missing telemetry itself triggers a diagnostic alert and no “zero failures” interpretation; critical safety mechanisms do not rely on one broken collector.
• Watch: missing series, alert evaluation, operator detection time and kill-switch effectiveness.

Side-effect duplication

• Inject retry/replay only into sandboxed notification/payment APIs.
• Expected: idempotency keys prevent duplicate actions and shadow/fallback routes remain non-authoritative when appropriate.
• Never replay real charges or irreversible business operations as a chaos test.

Practical experiment contract (illustrative YAML—not executable provider syntax)

YAML
experiment:
  id: inference-feature-brownout-001
  hypothesis: "isolated test traffic retains safe decision and deadline"
  environment: staging
  target:
    service: risk-inference
    tenant: synthetic-gameday
    release_digest: "sha256:<approved-release>"
  fault:
    type: feature_rpc_delay
    added_latency_ms: 100
    duration_seconds: 180
  safety:
    max_affected_rps: 40
    require_no_real_business_writes: true
    kill_switch_owner: "on-call-operator"
    stop_if_critical_invalid_decisions_gt: 0
    stop_if_eligible_bad_fraction_gt: 0.005
  expected_evidence:
    - edge_good_prediction_sli
    - feature_fetch_latency
    - fallback_mode_and_model_version
    - end_to_end_deadline_misses
    - fault_stop_and_full_recovery_timestamps

In a real tool, *enforce* experiment scope at multiple layers, not just as text in a file. Match targets precisely, validate authorization, connect monitored stop conditions to a tested abort action, limit start/end times, and verify restoration. This is a teaching schema, not a drop-in AWS FIS, Azure Chaos Studio or Kubernetes configuration.

How game days differ from automated chaos tests

• Narrow automation: Regular deterministic fault experiments detect regressions in circuit breakers, readiness, capacity, feature handling and telemetry.
• Game day: Includes on-call handoff, detection, triage, cross-team coordination, business approval, incident severity, status messages, customer-support preparation and restoration ownership.
• Tabletop first: Walk through a scenario without injecting a real fault when the failure mode is high risk or safety controls are unproven.
• Production-like before production: AWS Well-Architected advises validating fault injection and stop conditions outside production, then performing carefully planned drills with responsible teams.
• Exercise controls: Incident commander, SRE/model owner, feature-store owner, business owner, explicit halt authority and a communication channel separate from the impaired service.
• Learning output: Fix both system defects and unclear runbooks. A game day that reveals a broken alert or confusion over who may authorize failover is valuable even when no Pod crashed.

AWS Fault Injection Service provides fault experiment orchestration and CloudWatch-alarm-based stop conditions; Azure's chaos-engineering guidance calls for specific hypotheses, baselines, restricted blast radius and deliberate stop rules. Those platforms do not automatically determine if an ML prediction remains semantically safe; custom model/feature checks are still required.

Experiment readiness and ethical boundaries

• Obtain the relevant approvals for environment, business impact, customer exposure and data handling. Production experiments must obey incident/change policy.
• No experiment may send real duplicate payments, issue fake eligibility denials, violate feature freshness for consequential decisions, leak private requests or intentionally damage data.
• Prefer internal or synthetic entities; choose cohort-limited faults when live fidelity is essential.
• Never use unrestricted shared-network disruption without a tested path to halt it; shared feature services can affect unrelated models.
• Precheck staffing, support coverage, active incidents, release freeze, error-budget status, recovery capacity and explicit stop conditions.
• Treat a failed abort mechanism as a blocking test failure. Do not expand experiment scope until it is fixed.
• Verify that termination restores state; ending an experiment does not automatically reverse corrupted caches, duplicated messages, wrong model flags or broken routing.

How to verify an experiment result

Record separate measurements for:

• Before: typical traffic mix, workload, release digests, feature versions, latency, good-response SLI, fallback rate and known alarms.
• During: injected fault start/end timestamps, actual target coverage, customer-visible violations, model decision validity, side effects and guardrail triggers.
• After: recovered good-response fraction, stable primary/fallback behavior, queue drained or correctly rejected, feature freshness, actual route distribution and error-budget effect.
• Incident organization: alert arrival, acknowledge, diagnosis, decision authority, abort action, communications and operator recovery time.
• Correctness: A fallback returning HTTP 200 still needs a separately approved use-case contract; “zero 5xx” alone is not a pass.

Define objective pass/fail criteria before running. If the fault never reached the target, the experiment is inconclusive, not a resilience pass. If telemetry is missing during the critical window, report a measurement failure instead of claiming success.

Engineering trade-offs and limitations

• Staging-only experiments: Safer and repeatable but can miss production scale, real dependency quotas and genuine client behavior.
• Small production cohort: More realistic but may miss rare interactions; statistical confidence and blast-radius limits must both be reported.
• Synthetic traffic: Safer for business side effects; may underrepresent true feature skew, caches, tenant mix, and burst patterns.
• Scheduled game days: Improve human readiness; use skilled staff and consume engineering capacity.
• Continuous experiments: Catch regressions quickly but may create alert fatigue and require robust isolation; do not auto-run destructive tests.
• Single-fault tests: Easier to interpret; real outages can contain correlated failures, but compound tests should follow only after single-fault controls pass.
• Killing replicas: Simple infrastructure check; does not reveal silent stale-feature errors or incorrect model decisions.
• Alert-only exercise: Validates detection but not recovery; full drills also verify restored state and user outcomes.

Known limitations: Finite chaos tests cannot prove the absence of all failure modes. A successful test only supports its exact hypothesis, workload, version, measured scope and time window. Re-run after major model, dependency, feature or infrastructure changes and avoid generalizing one successful injection into a global reliability guarantee.

Common mistakes and production anti-patterns

• Injecting faults without a falsifiable user-facing hypothesis.
• Running faults in production before validating abort controls in staging.
• Leaving targets and stop conditions overly broad.
• Declaring success because Pods restarted, while customer predictions were stale or late.
• Sending copied requests into real notification or payment side effects.
• Using only HTTP 200/500 metrics and ignoring valid-prediction provenance.
• Failing to track sample denominators and missing telemetry during tests.
• Treating a small canary test as proof that rare critical cohorts are protected.
• Ignoring cross-model/shared-feature dependency contention.
• Ending an injection but not verifying queues, caches, routing and external state are restored.
• Assuming all required people will know the incident procedure without rehearsing it.
• Never converting discoveries into regression tests, updated runbooks and accountable remediation.

Compact production checklist

• Unique hypothesis, risk-ranked fault, immutable release identity and decision-type classification.
• Production-like staging trial with verified kill switch and trusted metrics.
• Explicit owners, affected users/tenants, duration, rate/replica scope and emergency stop authority.
• Baseline and steady-state SLI, prediction-validity and feature-freshness thresholds.
• Real-time stop monitoring that reacts to severe harm without waiting for a full statistical window.
• Independent confirmation that injected faults actually occurred and did not spread.
• End-of-test recovery verification and reconciliation of any effects.
• Auditable results, incident timeline, remediation tickets and scheduled retest.

Takeaway: A resilience design or dashboard is a *claim* about what will happen under failure. A safe, scoped chaos experiment and game day collect evidence that the full prediction path and its operators behave as expected, without sacrificing customer safety to make the exercise realistic.