How should confidence intervals and statistical uncertainty be estimated for production machine learning performance metrics?

A production ML metric is an estimate, not a fact about every future prediction. If a classifier is 80% accurate on 100 verified observations, that does not imply its long-run accuracy is exactly 80%. A confidence interval (CI) quantifies sampling uncertainty under explicit assumptions about data collection, independence, model selection and the target population. It does not correct biased labels, drift or leakage.

1. Which uncertainty are we estimating?

• Define the estimand: accuracy, positive-class recall, precision, log loss, calibration, or a *difference* between champion and challenger on a precisely defined production cohort.
• Record the metric's denominator, model version, event-time window, label maturity and selection mechanism. A CI calculated for 100 mature cases does not automatically represent all decisions if labels are selectively available.
• State the confidence level, usually 95%, and the method. In repeated comparable sampling under the method's assumptions, approximately 95% of constructed 95% intervals cover the fixed underlying population parameter. Do not say there is a 95% probability that the fixed parameter lies inside this particular frequentist interval.

2. Beginner-friendly numerical example: binomial accuracy

Assume 100 independent, representative, fully labeled predictions from one unchanged model, with 80 correct and 20 incorrect. Thus:

• Estimated accuracy p̂ = 80/100 = 0.80 (80%).
• A simple binomial plug-in standard error is SE = sqrt[p̂(1−p̂)/n] = sqrt[0.80×0.20/100] = 0.0400.
• For a two-sided 95% Wald (normal-approximation) CI, use z≈1.96: p̂ ± 1.96SE = 0.80 ± 0.0784 = [0.7216, 0.8784].
• Expressed as percentages, 72.16%–87.84%. The interval describes uncertainty from sampling; it is not a guarantee the next 100 predictions will have accuracy in that range.
• Wald intervals may undercover, behave poorly for small denominators and produce impossible values below 0 or above 1 near extremes. It is useful for learning, but generally Wilson or an appropriate exact/other interval is safer for proportions.

3. Wilson score interval for the same data

Wilson corrects some failures of the Wald approximation without needing to add arbitrary successes and failures.

• With z²=3.8416, compute denominator d = 1 + z²/n = 1.038416.
• The Wilson center is c = [p̂+z²/(2n)]/d ≈ 0.7889.
• The Wilson half-width is h = z sqrt[p̂(1−p̂)/n + z²/(4n²)] / d ≈ 0.0777.
• Wilson 95% CI = [c−h,c+h] ≈ [0.71117,0.86663], or 71.12%–86.66%.
• The center is not exactly p̂ because Wilson is derived by inverting a binomial score test. It generally has better coverage than the simple Wald interval, including near 0 and 1.
• Exact Clopper–Pearson is another alternative; it avoids relying on a normal approximation but is usually conservative, and its interval need not be as narrow as Wilson's.

4. Bootstrap interval from the same 100 observations

• Retain the 100 individual correctness indicators: eighty ones and twenty zeros.
• Randomly draw 100 indicators with replacement from these 100 values and calculate accuracy; repeat this 20,000 times.
• Take the 2.5th and 97.5th percentiles of the 20,000 resampled accuracy values. With NumPy's default_rng(247) and the sample ordered as eighty ones followed by twenty zeros, a reproducible illustrative result is [0.72, 0.87] = 72%–87%.
• This is the ordinary iid percentile bootstrap, not BCa, a studentized bootstrap, or a correction for delayed/selected labels. The exact empirical endpoints can vary with seed and resampling count.
• The bootstrap approximates uncertainty through resampling; it does not automatically solve biased observations, very rare positive cases, or dependence. Near extreme probabilities or tiny samples, an ordinary percentile bootstrap can give misleading or zero-width intervals.
• Compare results: Wald 72.16%–87.84%, Wilson 71.12%–86.66%, seeded percentile bootstrap 72%–87%. None is categorically correct under every data-generating process.

5. Reproducible Python

PYTHON
import numpy as np
from math import sqrt
from scipy.stats import binomtest

n, correct = 100, 80
p = correct / n
se = sqrt(p * (1 - p) / n)
wald = (p - 1.96 * se, p + 1.96 * se)
wilson = binomtest(correct, n).proportion_ci(
    confidence_level=0.95, method="wilson"
)

rng = np.random.default_rng(247)
outcomes = np.array([1] * 80 + [0] * 20)
boot = rng.choice(outcomes, size=(20_000, n), replace=True).mean(axis=1)
percentile = np.quantile(boot, [0.025, 0.975])
print("Wald:", wald)                # (0.7216, 0.8784)
print("Wilson:", wilson.low, wilson.high)  # ~0.71117, 0.86663
print("Percentile:", percentile)   # [0.72, 0.87]

For a huge dataset, avoid constructing one giant bootstrap matrix: sample in batches or implement a memory-efficient resampling procedure.

6. Precision, recall and rare-class uncertainty

• Precision uses TP/(TP+FP): the denominator is the number of *predicted positives*, not all observations.
• Recall uses TP/(TP+FN): the denominator is the number of *actual positives*. These are different conditional populations and require their own intervals.
• Example: if a fraud detector identifies 8 of 10 confirmed frauds, recall is 80%, but its 95% Wilson interval is much wider than for 80 of 100 because there are only 10 actual positives.
• Report confusion-matrix counts, mature-label coverage and subgroup sample sizes along with metrics. A 99%-accurate majority-class predictor may have near-zero fraud recall.
• Precision and recall can be correlated within a confusion matrix. F1, PR-AUC, ROC-AUC, cost-weighted loss and score calibration are not generally single binomial proportions; use appropriate influence-function, delta-method, or carefully designed bootstrap techniques rather than applying the accuracy formula indiscriminately.
• If a class has zero observed examples, saying “recall 0% ± 0%” or “100% certain” is wrong: recall may be undefined, and the positive-class data provide little or no useful information.

7. Minimum sample sizes and interval width

• There is no universal minimum n that guarantees a meaningful estimate; plan sample sizes around the desired CI width, expected rate, target effect size, class balance and dependence.
• Under iid Bernoulli sampling, a rough worst-case target half-width E for a 95% Wald approximation requires n ≈ 1.96² × 0.25 / E².
• For E=0.05 (±5 percentage points), n≈384.16, so round up to 385 independent representative outcomes. This is a rough *planning* approximation, not a promise of Wilson or bootstrap interval widths.
• Rare-positive recall needs sufficient positive outcomes specifically, not just large overall traffic. Correlation and selective labeling usually reduce the effective information.

8. Clustered, repeated, and time-correlated observations

• Many scores come from repeated users, devices, stores or time periods. Errors within one entity or incident can be correlated. Treating every row as independent often gives too-narrow intervals.
• For repeated customers, consider a cluster bootstrap: resample customers, retain all relevant observations in each selected customer and recompute the statistic.
• For time-dependent streams, consider a moving/stationary block bootstrap or another time-series-aware method that keeps meaningful local dependence. Block length and stationarity assumptions matter.
• For repeated users across time, choose a resampling design that acknowledges both clustering and temporal effects; no one bootstrap method handles all dependence.
• In overlapping evaluation windows, samples and alerts are not independent. Do not compare two correlated rolling-metric intervals as if they came from unrelated datasets.

9. Weighted metrics, delayed labels and label selection

• A naive binomial CI based on raw labeled n is generally invalid for propensity-weighted accuracy, since observations carry unequal weights and the propensity model may be estimated.
• From Question 246, Kish effective sample size gives a rough diagnostic of weight concentration: ESS=(Σw)²/Σw². It is not a universally valid substitution for n in a binomial CI.
• Use appropriate sandwich/influence-function standard errors or bootstrap a complete estimator, accounting for nuisance propensity/outcome model fitting and clustering. For cross-fitted estimators, respect the original cross-fit structure.
• From Question 245, delayed labels can create both smaller denominators and selective feedback. A narrower CI around observed cases cannot correct missing-not-at-random bias. State precisely which mature cohort is identifiable and what assumptions connect it to target traffic.
• When labels are censored or observed only after an intervention, define the estimand carefully; uncertainty calculations do not identify counterfactual outcomes under unsupported causal assumptions.

10. CI versus prediction interval, statistical significance, and monitoring

• A confidence interval expresses uncertainty about a population quantity such as mean accuracy, given a sampling model.
• A prediction interval targets a future random observation or outcome. For a future *count* of correct predictions, uncertainty includes the variability of that future count, and sometimes parameter-estimation uncertainty. It is not the same as a CI around current accuracy.
• Champion–challenger comparisons: on the same evaluation examples, compute paired differences and paired CIs; separate nonoverlapping 95% accuracy CIs are not a correct universal significance test.
• Repeated dashboards and multiple testing: checking dozens of metrics and segments every minute makes ordinary fixed-time 95% intervals unsuitable for uncontrolled repeated alerting. Account for family-wise error or false-discovery rates where appropriate; use confidence sequences / sequentially valid testing for continuous peeking.
• Alert thresholds: define a meaningful performance margin, min label volume, minimum coverage and a monitoring cadence in advance. An illustrative escalation is triggered when a statistically justified upper CI bound for error rate exceeds a predefined acceptable ceiling, after controlling the repeated-testing design.
• Production response: investigate data quality, label maturity, traffic mix and business impact before intervention. A significant decline in a well-defined metric is evidence to examine, not an automatic license to retrain.

Frequent errors

• Presenting the point estimate without counts or uncertainty.
• Calling a 95% frequentist CI a “95% probability range” for a fixed true metric.
• Using normal/Wald intervals for near-zero or near-one rates with tiny samples.
• Bootstrapping individual rows despite dependence by customer or time.
• Ignoring metric denominators for rare-class precision and recall.
• Reusing test sets for model tuning and reporting the resulting CI as though model choice were fixed in advance.
• Reporting many unadjusted CIs across segments and repeated time windows while treating each threshold crossing as conclusive.
• Confusing sampling precision with freedom from label-selection bias.

Curriculum distinction

• Question 246 derives IPW and doubly robust point estimators for selected labels; this lesson addresses uncertainty quantification for metrics and why weighted estimators need more than ordinary binomial standard errors.
• Question 245 discusses label delay and missing outcomes; this lesson specifies the validity limits of intervals built from partial feedback.
• Questions 242–243 establish temporal evaluation; this lesson concerns sampling uncertainty and alerting around the resulting metrics.

Key takeaway: A credible production metric should include its denominator, cohort, appropriate CI method, selection/dependence assumptions and alerting interpretation. Wilson is a solid default for independent binary proportions; robust resampling or influence-based inference is needed when samples are clustered, weighted or repeatedly monitored.

References

• NIST: Confidence intervals for a proportion
• SciPy: BinomTestResult.proportion_ci (Wilson and exact)
• UK public health technical guidance: Proportions and Wilson intervals
• SciPy: Bootstrap confidence intervals