How do you run trustworthy online A/B tests for production ML model versions?

An online ML A/B test randomly assigns eligible users or other independent business units to a stable model (control) or a candidate model (treatment), lets the assigned model make real decisions, and compares predeclared outcomes while protecting production with safety guardrails. Its purpose is to estimate *causal business impact*, not merely whether the model starts, runs quickly, or produces scores similar to the old version.

Prerequisite boundary: Question 121 covered staging tests; Question 122 covered canary release and rollback; Question 123 covered traffic mirroring with non-authoritative candidate predictions. A canary primarily limits blast radius, and a shadow test cannot reveal candidate-caused changes in customer behavior. Here the new skill is randomization, experiment validity, statistical interpretation and promotion decisions. Earlier experiment-tracking questions 76–82 concern training runs, not live randomized product outcomes.

Beginner-friendly example: delivery delay warnings

A delivery platform currently uses approved model r27 to decide whether checkout should show a “may be late” warning. Candidate r28 passed staging, a safe limited rollout and shadow tests, but its score thresholds differ. Does r28 actually reduce late-delivery complaints *without* hurting checkout completion?

• Population: Eligible customers beginning checkout during an agreed experiment period.
• Assignment unit: Customer account, not individual API call. All eligible sessions for the same customer get the same variant to avoid switching policies on refresh.
• Control A: Existing r27 decision policy.
• Treatment B: New r28 decision policy; both paths have separately pinned runtime and feature schema.
• Primary outcome: Percentage of eligible assigned customers completing checkout within a predeclared attribution window.
• Secondary outcome: Late-delivery complaint rate among orders placed, interpreted cautiously because treatment may change who orders.
• Guardrails: Severe checkout failures, p95/p99 serving latency, feature freshness, complaint escalation and refund cost.
• Analysis plan: Define duration, sample-size target, minimum detectable effect, metric denominator, exclusions, delayed outcomes and decision criteria *before* reading the result.

The difference between variants must be controlled and logged. If r28 also changes the UI layout, the measured effect is of the combined change, not the model alone.

Step-by-step production experiment

1. Write hypothesis and success criterion. Example: r28 improves customer-level checkout conversion by at least a meaningful amount while not breaching latency or complaint guardrails. Choose one primary metric and designate exploratory subgroup analyses.
2. Define eligibility and randomization unit. Randomize customers, merchants, drivers, geographies, or clusters depending on interference. If one customer's assignment affects another user's supply or price, simple user-level randomization can violate independence.
3. Assign once and persist. Use an experiment service or deterministic salted hash of a stable privacy-safe ID, experiment ID and allocation scheme. Never hash the live request timestamp or re-randomize for every model call. Protect IDs and salt.
4. Freeze variant releases and policy. Link A/B assignment to exact model digests, preprocessing, feature schema, threshold, configuration and environment. Changes during the trial require explicit versioned experiment epochs or a restart.
5. Separate exposure from inference. Log assignment, *actual exposure*, model invocation, response and eventual business outcome with immutable experiment and variant IDs. Users assigned but not exposed are not the same population as exposed users; analyze using a precommitted estimand.
6. Run an A/A and routing-integrity check. Before trusting model uplift, confirm assignment logging, cohort parity and expected traffic shares; check sample ratio mismatch (SRM), missing outcomes and duplicate events. Test both branches with the same model if useful.
7. Begin with a safety-limited ramp. A small production share can detect egregious regressions; do not infer causal lift from an underpowered safety ramp. Define experiment duration and power separately from Question 122's release progression.
8. Monitor while running. Automatically shut down serious safety breaches. Regularly inspect logging health, eligibility counts, SRM and hard guardrails; repeatedly applying an ordinary fixed-horizon significance threshold without correction can inflate false positives.
9. Wait for the specified outcome window. Delayed labels, order completion and complaint events need event-time attribution, late-arrival handling and a predeclared maturity rule.
10. Analyze and decide. Estimate treatment minus control effect, uncertainty and business importance using a method suited to the randomization unit and planned testing strategy. Document whether to promote, extend via a valid design, reject or investigate.
11. Retain rollback and auditing. Promotion to 100% does not undo decisions from the experiment. Archive assignment policy, variant identities, treatment effect, uncertainty, segment safety reviews and approval.

Numerical example 1: exposure and conversion uplift

Suppose 20,000 eligible customers are randomized 50/50 and each is counted once after their outcome window matures.

• Control r27: 10,000 customers; 1,000 complete checkout → 10.0% conversion.
• Treatment r28: 10,000 customers; 1,120 complete checkout → 11.2% conversion.
• Absolute lift: 11.2% − 10.0% = 1.2 percentage points.
• Relative lift: 1.2 / 10.0 = 12%.
• Under a simple independent-binomial normal approximation, standard error of the absolute difference is

sqrt(0.10×0.90/10000 + 0.112×0.888/10000) ≈ 0.00435, or 0.435 percentage points.

• Approximate 95% interval for the observed difference: 0.012 ± 1.96×0.00435 ≈ [0.0035, 0.0205], or +0.35 to +2.05 percentage points.
• Approximate z-score: 0.012/0.00435 ≈ 2.76.

Interpretation: In this simplified preplanned, independently randomized example, evidence supports a positive treatment effect; it does not establish that every region improves, that the effect will persist indefinitely or that guardrails pass. If allocation is clustered, repeated measurements are correlated, or the experiment was stopped after repeatedly checking p-values, use the appropriate design-based or sequentially valid analysis.

Numerical example 2: sample ratio mismatch (SRM)

A service is configured for a 50/50 assignment across 20,000 unique users, but telemetry reports 10,800 control and 9,200 treatment.

• Expected counts = 10,000 per arm.
• Chi-square statistic (10800−10000)^2/10000 + (9200−10000)^2/10000 = 64 + 64 = 128 with one degree of freedom.
• This is overwhelmingly inconsistent with the intended 50/50 split under simple independent random assignment (far beyond a 3.84 reference threshold for p≈0.05).
• Decision: STOP and investigate measurement/routing validity before interpreting uplift.

Potential causes: bucket hashing bug, eligibility checked after assignment, variant-specific crashes, redirects that lose IDs, logging loss, bot filtering, imbalanced rollout region, or a comparison restricted to users who completed an outcome. SRM should normally be checked on the assignment population, with exposure and outcome counts inspected separately. Changing the denominator after seeing the effect can introduce selection bias.

A milder 50.4/49.6 split could be plausible at small sample sizes; use a statistical SRM check rather than treating every unequal count as a defect.

Numerical example 3: minimum detectable effect and power

Suppose baseline conversion is 10%, and the smallest worthwhile improvement is +1.0 percentage point (to 11%). For a simple two-sided 5% test at approximately 80% power, a common normal-approximation planning formula is

n_per_arm ≈ 2 × (1.96 + 0.84)^2 × 0.10 × 0.90 / (0.01)^2 ≈ 14,112 customers per arm.

Roughly 28,224 unique customers are needed in total under these simplified assumptions. Attrition, delayed outcomes, noncompliance, cluster randomization, unequal allocation, multiple comparisons and variance adjustment can materially change the requirement. Confirm with a production-grade power calculation or simulation for the actual estimand.

If eligible traffic is 4,000 unique customers/day with 50/50 allocation, raw recruitment alone takes 28,224/4,000 ≈ 7.1 days; a design requiring full weekly-seasonality coverage and 7-day outcome maturation needs a longer calendar window. More request traffic does not substitute for more independent *users* when user is the assignment unit.

Why traffic weight is not experimental randomization

A managed endpoint may distribute, say, 10% of *requests* to a production variant. That offers rollout exposure control (Question 122) but not necessarily unbiased customer-level A/B assignment. A frequent shopper might generate 50 requests while another generates one; per-request routing can send both users across arms, contaminate their experiences and overweight frequent traffic.

For causal evaluation, assign at the correct experimental unit, hold assignments stable, and then route to the pinned variant. Amazon SageMaker AI documents production variants with weighted distribution and explicit TargetVariant selection; using a stable experiment allocator to select a variant is an application-level design, not a guarantee furnished by endpoint weights alone.

Outcome attribution and delayed labels

• Record the first eligible exposure, unique assignment ID, model/policy version, relevant decision and source timestamps.
• Set a fixed outcome window, for example conversion within 24 hours and complaints within 7 days. Avoid comparing mature control labels with immature treatment labels.
• Deduplicate retries, repeated events and multi-device identities according to a defined unit policy.
• Decide whether *intention-to-treat* (analyze all assigned eligible units) or an exposure-based estimand is primary. Conditioning on post-assignment behavior can bias results.
• Distinguish business decision uplift from traditional offline prediction accuracy. A model with a better AUC can still harm customer experience through changed interventions or thresholds.
• Watch interference and feedback: the treatment may affect the population's behavior, data collection, inventory and what is later used to train models.

Hard safety guardrails versus statistical success

Immediately stop or route to stable when there is a critical authorization failure, harmful business action, severe customer impact, invalid response schema, or sustained outage. Operational safety is more important than completing a perfectly powered experiment.

Hold / investigate if experiment SRM, missing labels, version-assignment mismatch, feature staleness or biased data capture makes the estimated effect untrustworthy.

Promote only when required metrics, uncertainty and business value criteria pass, no essential guardrail fails, and the trained candidate is still the exact version that ran in the experiment. A positive but tiny uplift can be uneconomic if serving cost or support burden is large.

Monitor: assignment/exposure counts by arm, routing fidelity, user-level eligibility, SRM p-value, missing-outcome rate, outcome maturity, conversion and cost, p95/p99 latency, 5xx/timeouts, feature freshness, traffic mix, model score distribution and guardrails by meaningful cohorts. Define clear owners for experiment platform, model endpoint, business metrics and incident response.

Failure scenarios and debugging runbooks

Control users unexpectedly see candidate behavior

• Inspect unstable IDs, per-request routing, cache key omissions and fallbacks that ignore experiment assignment.
• Freeze the experiment, restore deterministic routing, and invalidate contaminated observations unless a principled analysis is possible.

The observed 50/50 sample becomes 54/46

• Audit eligible assignment counts by region and day, traffic allocation, exposure logging and variant-specific request failures.
• Diagnose SRM *before* evaluating the headline treatment effect. Never “correct” a broken randomization just by reweighting counts without understanding why it broke.

Business uplift looks strong but model errors rise

• Check user impact, per-arm latency, timeouts, model fallback rates and whether the experiment altered the customer mix. Apply predeclared critical guardrails even if conversion appears better.

Labels arrive unevenly across arms

• Inspect time-to-label distribution, event ingestion delays, right censoring and outcome windows. Do not treat unobserved delayed labels as negative outcomes.

A p-value falls below 0.05 on day two, then disappears

• Diagnose repeated significance peeking and the initial power plan. Follow the predeclared fixed horizon or implement alpha-spending/confidence sequences designed for sequential monitoring, rather than opportunistically ending at the most favorable look.

New and old models use different feature definitions

• Compare versioned feature contracts, timestamps and missing-value behavior. If the change is intentional, characterize the full treatment policy; if accidental, correct and restart with a new documented experiment epoch.

Practical implementation pseudocode

PYTHON
from hashlib import sha256

def choose_arm(experiment_id: str, stable_user_id: str, salt: str):
    key = f"{experiment_id}:{stable_user_id}:{salt}".encode()
    bucket = int.from_bytes(sha256(key).digest()[:8], "big") / 2**64
    return "treatment" if bucket < 0.5 else "control"

# Persist the arm and the exact release/version in a dedicated
# assignment log BEFORE invoking the chosen serving endpoint.
# Do not use user-supplied IDs as proof of authenticated identity.
# Production services must protect assignment salts and IDs.

The hash creates stable approximate allocation for sufficiently well-distributed identities; it is not a replacement for an experiment service's eligibility, persistence, audit, privacy and causal-design controls. Test the unit-level assignment balance and end-to-end exposure logging.

Common production mistakes

• Treating 50/50 request weights as stable randomized customer assignment.
• Running shadow models and claiming that business uplift was causally measured.
• Selecting a winning metric after looking at outcomes; ignoring multiple testing.
• Repeatedly peeking at fixed-horizon p-values and stopping at the first favorable result.
• Ignoring sample ratio mismatch, missing events or post-treatment selection.
• Failing to reserve time for delayed labels or seasonality.
• Mixing changing model versions in one arm without explicit experiment epochs.
• Averaging over regions when an important segment has severe harm.
• Assuming statistical significance guarantees commercially meaningful improvement.
• Forgetting rollback and compensation for real treatment-side decisions.
• Using offline ground-truth accuracy as a substitute for randomized customer impact.

Production experiment acceptance checklist

• Explicit randomized unit, eligibility, allocation, stable IDs and assignment audit.
• Pinned model and serving release for each arm with valid feature contracts.
• One primary outcome, estimation method, minimum worthwhile effect, sample-size plan, windows and guardrails.
• A/A checks, SRM monitoring, per-arm observability and missingness tracking.
• Safe stop/rollback automation and accountable decision owners.
• Mature outcomes, uncertainty, sensitivity analyses and privacy-respecting result retention.
• Reproducible go/no-go decision, including effects on cost, user experience and model quality.

Takeaway: Shadow traffic tells you whether a new model can *run* safely on live inputs. A well-designed online A/B experiment tells you whether using that model to make real decisions *improves the business outcome*. Randomize the right units, verify assignment and data quality, precommit the analysis, and treat statistical evidence and safety guardrails as separate promotion requirements.