How should production ML teams govern changes to model-quality SLO definitions, targets, cohort scopes, and measurement policies without masking historical reliability failures?

# How should production ML teams govern changes to model-quality SLO definitions, targets, cohort scopes, and measurement policies without masking historical reliability failures?

Short answer

Version the measurement contract and change it prospectively; never overwrite the metric definition that governed an earlier reporting window. A change from 95% to 90% recall, or a change from all eligible frauds to only reviewed high-confidence frauds, may make a dashboard green without improving a single customer outcome. Model-quality governance therefore needs immutable historical SLO/SLI contracts, parallel old/new reporting, evidence about target-population and label changes, stakeholder/risk approval, prospective activation, clear migration and a revised error-budget/release-policy mapping. A genuine correction to faulty historical data may require a separately labeled restatement, but must not silently erase the original finding.

1. Treat the quality SLO as a versioned production contract

For every SLO version, store:

• Service and decision: production model or decision policy, intended customer harm, product scope, approved use case and accountable team.
• SLI semantics: numerator, denominator, threshold direction, positive-class definition, exclusion rules, metric aggregation, cohort predicates, weighting and risk/severity basis.
• Measurement implementation: prediction-event timestamp, served model/feature version, label-source contract, label-availability horizon, adjudication status, missing-label assumptions, data-query/code digest and monitoring owner.
• SLO policy: numerical target, rolling or calendar window, evidence-quality requirements, severity exceptions, budget and burn interpretation, escalation/release-freeze rules, fallback and override process.
• Governance metadata: policy version, author, reviewers, approved effective date, next review, approval ticket, linked change request and prior superseded version.

A stable identifier should be something like fraud_recall_prod:v1, fraud_recall_prod:v2. Metric names alone cannot determine comparability. Two panels both labeled "fraud recall" can be fundamentally different if one excludes delayed cases.

2. Numerical example: proposed recall target and denominator change

Suppose the approved v1 contract is:

• 28-day production recall on all eligible, mature, independently confirmed frauds.
• SLO at least 95%.
• A completed period has 1,000 eligible confirmed frauds, of which the model correctly flagged 920 and missed 80.
• v1 recall = 920 / 1,000 = 92%. Allowed misses under this completed-window v1 objective = 50; observed misses = 80, i.e. 30 beyond allowance. The period failed v1.

The product team proposes changing v2 to 90% recall, with the same population, label policy and window:

• Reinterpreting the historical 92% rate against a 90% proposed target would show a pass, but that *does not retroactively change the fact that the previous period failed the approved 95% v1 contract*.
• At the same 1,000 positives, the 90% target would allow 100 misses rather than 50. This is a substantive increase in tolerated missed frauds; the risk owner must assess customer costs and compensating controls, not merely developer delay from freezes.
• A prospective, explicitly approved lower target could be defensible only if evidence shows the old threshold was unachievable or poorly aligned with user harm, and other safety constraints remain satisfied. It is not an acceptable emergency way to wipe an incurred breach.

Second proposal: alter the denominator

• Of the same 1,000 confirmed frauds, 800 were cases with prompt human-review confirmation; assume 760 were correctly flagged and 40 missed in that subset.
• The remaining 200 mature confirmed frauds were harder or received slower independent adjudication: 160 correctly flagged, 40 missed.
• Across *all* eligible cases, the original 920/1,000=92% remains unchanged.
• If v2 evaluates only prompt-reviewed cases, its reported recall becomes 760/800=95%. The 95% target *now appears to pass*, but 40 missed frauds among excluded cases did not disappear.
• The excluded 200 have 160/200=80% recall, worse than the included 95%. This is a concrete selection-bias red flag: improving the metric by removing a difficult cohort can conceal greater harm.
• If instead a legitimate product change truly makes those 200 cases *out of scope for the decision system*, document the service/ownership transfer and ensure their customer risk is tracked elsewhere. It remains improper to rewrite the original v1 service population retroactively.

Correct decision in this incident

1. Preserve the completed window as v1: FAIL, recall 92%. Retain its incident, release decisions, customer harm and audit record.
2. Record both proposals as distinct changes: threshold relaxation and population/label-scope revision. Do not blend them into one ambiguous "quality improvement."
3. Reject silent exclusion of the slow-reviewed 200 cases without legitimate scope change, validated alternative measurement and risk approval. The 80% excluded-slice recall deserves separate attention.
4. If the risk/product/engineering owners eventually approve a 90% v2 threshold, activate it prospectively from a specified future window; display v1 and v2 metrics side by side in a dual-reporting phase.
5. Revalidate SLO burn alerts, release-freeze gates, critical subgroup protections and the model-quality consequences before switching enforcement.
6. Even after v2 becomes active, previous contractual compliance and rollback/postmortem evidence must continue to reference v1.

3. What kinds of change need review?

• Threshold/target change: 95%→90% recall or a different allowed cost-weighted loss. Changes tolerated user harm but not necessarily the measured data.
• Denominator change: all frauds→reviewed frauds; all new users→only app users who opted into extra verification. This changes the population being represented.
• Numerator/label definition: what counts as correctly detected, positive fraud, appeal overturn, confirmed false positive or matured outcome.
• Window change: rolling 28 days→calendar month; event-time→label-finalization-time. Can alter which failures fall into a period and when results become official.
• Aggregation change: micro overall recall→macro cohort recall, traffic weights→fixed reference weights, raw→adjusted labels. Ratios may no longer be directly comparable.
• Measurement implementation: correcting a join, routing model-ID provenance, event-time feature lookup or delayed-label processing can change historical calculations without changing the *intended* user-facing definition.
• Cohort rules: geographical eligibility, protected/critical subgroup criteria, class taxonomy, inclusion/exclusion changes or product migration.

Each requires a documented semantic diff: what changed, why, which historical values differ and which customers are newly covered or excluded.

4. Distinguish policy redesign from a data-quality correction

• A policy change updates what "good enough" means. Apply it from an approved effective date; do not convert the prior failure to success.
• A measurement defect fix corrects an implementation that demonstrably miscomputed the previously agreed measure, such as assigning labels to the wrong model IDs. A historical restatement may be warranted, but publish both the *original reported result* and *corrected restated result*, with a reason, lineage, approval and date.
• A target population genuinely changed because the product no longer serves some requests. Create a new scope version, transfer ownership of excluded harms and evaluate continuity. Don't discard outstanding adverse cases as mere denominator noise.
• A redefined label from revised adjudication policy may be non-comparable with earlier labels; dual-label a representative bridge sample if ethical, legal and feasible. Where bridging is impossible, annotate a series break rather than drawing a smooth improving trend.

5. Migration and dual reporting

Before activating a new v2:

• Build a mapping inventory showing which historical and future events are eligible under each version. Keep original prediction and label provenance so both policies can be evaluated.
• Backtest v2 on frozen, event-time-correct historical data, without changing what the historic v1 contract said. Report old rule applied to old data, new rule applied to same bridge data and new rule on newly observed data as distinct comparisons.
• For an illustrative 28-day rolling-window metric, consider running old and new panels concurrently for at least one complete comparable window plus the necessary label-maturity horizon, subject to system risk and available data. This is a context-specific migration choice, not a universal number of days.
• Mark the enforcement transition clearly; specify whether a window straddling the cutover is evaluated under one rule, split into version-specific subwindows, or deferred until enough v2 exposure matures.
• Put a visible policy version and "not directly comparable" annotation on plots and exported datasets. Never splice ratios from different denominators into an unlabeled single trend.
• Snapshot historical query, label contract and dashboard configuration, so an auditor can reconstruct the decision as it was made.

6. Governance and approval path

• Model owner: confirms business task, class-specific metric semantics, technical feasibility and impact on true positives/false negatives.
• Ground-truth/data owner: verifies eligibility, label maturity, source revisions, selected observations, lag and reconstruction accuracy.
• SRE/release owner: checks how the revised budget changes freezes, pages, burn alerts, canary decisions and recovery gates.
• Product owner: owns user-experience and innovation/reliability trade-offs.
• Risk, fairness, privacy or safety reviewer: scrutinizes critical or excluded populations and hard non-negotiable protections.
• Approval authority: named cross-functional approver(s) approve an effective date and residual risk; exceptions require time-bounded documented authorization.

The change request includes *old and new contracts*, the motivation, backtest and subgroup results, the impact on historical compliance, forecasted release-policy changes, residual unknowns, dual-reporting schedule, monitoring plan, rollback and explicit sign-off. The policy is only considered changed when both the approved contract and actual dashboard/enforcement implementation reflect the new version.

7. Selective outcomes and delayed ground truth

• In fraud, credit, clinical triage and content enforcement, the system's actions may affect which outcomes are reviewed and eventually labeled. The reviewed-only denominator can be selected by the model itself.
• Track eligible decisions, confirmed outcomes, unresolved and censored outcomes, label lag by version/slice and adjudication rule. Excluding a hard delayed-label cohort may inflate recall without reducing misses.
• Avoid calling unknown predictions "correct." If only a subset is labeled, state the estimand; appropriate audited adjustment may be needed and cannot magically recover unobservable ground truth.
• Freeze the event-time mapping from prediction to actual served model and label horizon. Backfills and new labels should create provenance-tracked revisions rather than overwrite evidence silently.
• For safety-critical groups with little sample support, use specialist checks, risk limits and conservative fallback; do not mask uncertainty with large aggregated cohorts.

8. Historical reporting, auditability and risk exceptions

• Keep immutable approved v1 policy ID, original reported outcome, budget state, triggered freeze/incident, approvals, and subsequent v2 change record.
• A restated historical metric should include the original and corrected values, correction reason, query/label versions, scope, author and approval, without quietly deleting breach history.
• Record which release decisions would have changed under v2 and flag that as a counterfactual analysis, not proof that v1 decisions were wrong or customers would have been safer.
• Safety, fairness and privacy constraints can be stricter than the aggregate SLO. A lower aggregate recall objective must not automatically weaken rare severe-group guardrails or justified emergency response.
• New versions must carry an audit-traceable model/feature/label/policy bundle, so later incidents are evaluated under the rule active at the time.

9. Integrate the change into live release governance

• Update alert thresholds and burn calculations without confusing old-budget and new-budget units. Rehearse decisions on representative historical events; verify that serious missed frauds still trigger correct escalation.
• Shadow the proposed v2 policy next to active v1 when safe. Log disagreement cases (v1 freezes, v2 allows; v1 passes, v2 blocks), and review their actual and hypothetical risk before enabling v2 enforcement.
• Stage changes to release governance, not just model code: ensure on-call runbooks, CI gates, dashboards, incident templates and escalation channels use the same policy version.
• Revalidate the "reopen release" criteria after a freeze and protect verified reliability repairs and urgent safety/security fixes through approved exception paths.
• If the new rule results in harmful releases, misleading label statistics or lost critical-cohort visibility, revert the enforcement policy version with approval; preserve both versions' history. Reverting a policy need not change the actual predictive model.

10. Failure scenarios and practical trade-offs

• Metric laundering: lowering recall from 95% to 90% purely to turn a red graph green makes no model improvement.
• Denominator gaming: restricting evaluation to easy reviewed cases hides 40 missed frauds in the numerical example.
• Policy/measurement conflation: a backfilled corrected label is not equivalent to a governance decision to tolerate more errors.
• Window reset: moving from 28-day rolling to a new calendar month can visually erase a failure if cross-window context is not retained.
• Incomparable charts: smooth joined trends with different cohorts, labels or weights can manufacture artificial improvement.
• Dashboard/policy mismatch: approving v2 without migrating release gates can create contradictory automatic freezes or unsafe approvals.
• Permanent dual reporting: two dashboards cost compute, analyst time and on-call attention; phase out enforcement duplication after successful verification while retaining archived history.
• Overly rigid governance: changing a genuinely defective metric may be necessary; allow a controlled, transparent restatement instead of requiring everyone to keep using a known-broken number.
• Insufficient power/coverage: tiny subgroup recall measurements may not support strong claims; keep uncertainty visible and separate hard safety protections.
• Changing targets after an outage: unapproved retroactive SLO relaxation destroys trust, comparability and error-budget accountability.

11. Production checklist

• Freeze the old contract: approved SLI specification, thresholds, period, label code/data snapshots, current incident and enforcement status.
• Write a semantic diff: old→new numerator, denominator, cohort, label, window, model/feature coverage, reason and trade-offs.
• Simulate both policies: same eligible bridge data, subgroup counts, uncertainty, historic pass/fail and release-action changes.
• Protect excluded users: prove every removed cohort has legitimate scope transfer and an owner for its quality/safety needs.
• Approve: product, model, SRE, data and applicable risk/safety reviewers sign off on effective date, migration and exception policy.
• Dual report: old and new as separate versioned series until sufficient comparable matured outcomes support cutover.
• Integrate: update dashboards, SQL/metric code, on-call alerts, release freezes and audit/recovery gates together.
• Preserve history: show v1 failed 92% vs required 95%; later v2 compliance is prospective and cannot erase that fact.
• Review: monitor customer harm and decision changes; revert/adjust v2 if it misrepresents service risk.

Curriculum distinction

• Q259 defines ML-specific quality SLIs, SLOs, denominators and budgets; this lesson is not a redefinition tutorial.
• Q260 uses agreed budgets to decide which releases can proceed. Q261 audits and tunes the performance of that release-decision policy. Q262 governs changes to the underlying quality contract itself, especially population, labels and measurement meaning, with immutable historical accountability.
• MLOps Q130 concerns general inference observability and incident response; this is specifically about versioned, approved changes to the semantics of a label-based model-quality SLO.

Key takeaway: Changing what “good model quality” means may be justified, but only with prospective policy versions, old/new bridging, explicit customer-risk approval and preserved historical failures. A higher reported metric is not improvement when the metric silently stops counting the people the model was harming.

References

• Google SRE Workbook: Implementing SLOs
• Google SRE Workbook: Error Budget Policy
• Google SRE Book: Service Level Objectives
• Google SRE Workbook: Example SLO Document