How can inverse-propensity weighting and doubly robust estimation address selection bias when evaluating machine learning models with selectively observed labels?

Inverse-propensity weighting (IPW) and doubly robust (DR, or augmented IPW/AIPW) estimation are tools for evaluating a model when some prediction outcomes are observed more often than others. They aim to estimate accuracy or mean loss across all eligible decisions, not just the biased subset whose labels happen to be known. Their guarantees depend on assumptions about *why labels are missing*; neither method can manufacture information for a group whose labels are never observed.

A new question beyond delayed-label handling

• Question 245 establishes label maturity, coverage and the danger of treating unknown outcomes as correct or incorrect. This lesson asks a more specific question: how should already observed outcomes be reweighted or augmented when their probability of observation differs systematically?
• Define Aᵢ = 1 if the fixed model's prediction for example i was correct and 0 otherwise. Let Rᵢ=1 if that correctness can be verified from an observed trustworthy label; otherwise Rᵢ=0. All N predictions have their own recorded features Xᵢ, model version and decision; Aᵢ is observed only when Rᵢ=1.
• The target is population accuracy θ = E[A] for the *specified production cohort under its actual decision policy*, not merely observed-case accuracy.

Step 1: Observation propensity, not a model's predicted probability

• Let πᵢ = P(Rᵢ=1 | Xᵢ) be the probability that a case's label is observed under the relevant observation/review mechanism. This label-observation propensity is not the model's predicted probability of fraud.
• A case with π=.20 is expected to be labeled only 20% of the time among comparable cases. If it *is* observed, its IPW weight is w=1/.20=5, representing underobserved similar cases.
• Whenever a selection mechanism is known from randomized audit sampling, record its actual design probabilities. Otherwise fit and calibrate a propensity model using features genuinely observed for *all* decisions, including whether their labels were obtained.
• The key requirement is missing at random conditional on X: after conditioning on recorded variables, the correctness/outcome for a case does not independently determine whether it is labeled. This assumption cannot be validated solely from observed labels.

Step 2: Basic observed-only and weighted estimators

• Observed-only accuracy: θ_observed = Σ(Rᵢ Aᵢ) / Σ Rᵢ. This can be biased because the observed population may disproportionately represent easy, difficult, risky or frequently audited cases.
• Horvitz–Thompson-style IPW accuracy: θ_HT = (1/N) Σ Rᵢ Aᵢ/πᵢ. For known, correctly specified positive observation probabilities, this is an unbiased estimate of the finite-cohort mean under the observation sampling mechanism; a finite-sample realization may be outside [0,1] for bounded metrics.
• Self-normalized (Hájek) IPW accuracy: θ_SNIPW = Σ Rᵢ Aᵢ/πᵢ / Σ Rᵢ/πᵢ. It stays within [0,1] for binary accuracy with positive weights, usually has useful finite-sample behavior, but is not exactly unbiased at finite N.
• Effective sample size (Kish-style diagnostic): ESS = (Σ_observed wᵢ)² / Σ_observed wᵢ². It gauges weight concentration among labeled cases; it is not proof of representativeness or a complete confidence-interval calculation.

Step 3: Beginner-friendly numerical example with unequal label selection

Consider 100 fraud-classifier predictions belonging to two risk strata, with historical or designed observation probabilities known:

• High-risk stratum: 20 total predictions, observation probability π_H=0.50, 8 actually labeled and 5 correctly classified among those 8. Each observed high-risk case receives w_H=1/0.50=2.
• Low-risk stratum: 80 total predictions, observation probability π_L=0.25, 24 actually labeled and 20 correctly classified among those 24. Each observed low-risk case receives w_L=1/0.25=4.
• Actual labeled cases: 8+24=32 (coverage 32%). Observed correct predictions: 5+20=25. Hence naïve observed-only accuracy = 25/32 = 78.125%.
• The observed counts 8 and 24 need not equal their propensities times total population counts (10 and 20 respectively). Selection is random; realized samples fluctuate. This is why raw and normalized IPW can differ.

Calculate the IPW numerator and denominator:

• Weighted correct count = 5×2 + 20×4 = 10+80 = 90.
• Weighted number of observed cases = 8×2 + 24×4 = 16+96 = 112.
• Horvitz–Thompson IPW = 90/100 = 90.00%; its fixed denominator is all 100 decisions.
• Self-normalized IPW = 90/112 ≈ 80.357%, with normalization by realized observation weights.
• The two IPW variants differ because 112 estimated population equivalents happened to appear from the 32 observed labels, versus the known actual N=100. This does not mean either number is guaranteed to match the unknown full-data accuracy.

Calculate ESS to expose weight concentration:

• Sum of squared weights = 8×2² + 24×4² = 32+384=416.
• ESS = 112²/416 = 12,544/416 ≈ 30.15, compared with 32 actually labeled examples.
• ESS would shrink more sharply if just a few cases received extreme inverse propensities. Report both raw labeled n and ESS; neither substitutes for a valid selection mechanism.

Step 4: Doubly robust (AIPW) estimator with an outcome model

A DR estimator uses two independently defined nuisance models:

• The observation model π̂(X) ≈ P(R=1|X).
• An outcome-regression model m̂(X) ≈ E[A|X], which predicts the probability that the fixed evaluated classifier will be correct. This is a model of evaluation correctness, *not necessarily the classifier's own predicted class probability*.

For N target-cohort decisions, the simple missing-outcome AIPW estimator is: θ_DR = (1/N) Σ [m̂(Xᵢ) + Rᵢ/π̂(Xᵢ) × (Aᵢ − m̂(Xᵢ))].

• The first term estimates correctness for every decision, including the unlabeled cases.
• The second term uses weighted residuals for genuinely observed outcomes to correct the estimated average.
• Under conditional missing at random, positivity, regularity and appropriate estimation (e.g. cross-fitting for flexible nuisance learners), this estimator is consistent when either the observation model or the outcome regression is correctly specified, even if the other is misspecified. It is not a guarantee that a small-sample estimate is correct.

Continue the same 100-case example:

• Suppose an independently trained outcome model estimates m̂_H=0.60 for each of the 20 high-risk cases, and m̂_L=0.80 for each of the 80 low-risk cases.
• Baseline model-implied correct count = 20×0.60 + 80×0.80 = 12+64=76; model-only accuracy estimate 76/100=76%.
• High-risk observed-residual sum = 5 − 8×0.60 = 0.20. Correct by inverse propensity: 0.20/0.50=0.40.
• Low-risk residual sum = 20 − 24×0.80 = 0.80. Correct by inverse propensity: 0.80/0.25=3.20.
• DR estimated correct count = 76+0.40+3.20=79.60.
• Doubly robust accuracy = 79.60/100 = 79.6%.
• Different estimators tell different stories on one small sample: naïve 78.125%, HT 90.0%, self-normalized IPW 80.36%, outcome regression 76.0%, DR 79.6%. Without knowing the 68 missing outcomes, we cannot declare one numeric estimate the ground truth; compare estimator assumptions, calibration and uncertainty.

Reproducible Python numerical calculation

PYTHON
N = 100
strata = [
    {"total": 20, "labeled": 8, "correct": 5, "pi": 0.50, "m": 0.60},
    {"total": 80, "labeled": 24, "correct": 20, "pi": 0.25, "m": 0.80},
]

observed_n = sum(s["labeled"] for s in strata)
observed_correct = sum(s["correct"] for s in strata)
weighted_n = sum(s["labeled"] / s["pi"] for s in strata)
weighted_correct = sum(s["correct"] / s["pi"] for s in strata)
squared_weights = sum(s["labeled"] / s["pi"]**2 for s in strata)

naive = observed_correct / observed_n
ht_ipw = weighted_correct / N
sn_ipw = weighted_correct / weighted_n
ess = weighted_n**2 / squared_weights
dr = (
    sum(s["total"] * s["m"] for s in strata)
    + sum((s["correct"] - s["labeled"]*s["m"]) / s["pi"] for s in strata)
) / N
print(f"Naive={naive:.5f} HT={ht_ipw:.5f} SNIPW={sn_ipw:.5f}")
print(f"ESS={ess:.2f} DR={dr:.5f}")
# Naive=0.78125 HT=0.90000 SNIPW=0.80357
# ESS=30.15 DR=0.79600

Critical assumptions and failure scenarios

• Conditional ignorability / missing at random: if human investigators are more likely to label truly fraudulent cases *even after controlling for X*, both IPW and basic DR may be biased. Include relevant logged review signals and estimate sensitivity to plausible missing-not-at-random mechanisms; do not claim to observe or prove the assumption.
• Positivity / overlap: every covariate or action subgroup in the evaluation target must have a positive chance of being labeled. If rejected applicants are *never* approved, their repayment outcomes are undefined/unobserved under that policy; weighting cannot identify their counterfactual outcomes.
• Policy-dependent outcomes: when an action changes the outcome itself (loan approval, medical treatment, offer targeting), explicitly specify the *potential outcome and target policy* being evaluated. Correct treatment-policy off-policy estimators require action propensities, consistency and additional causal identification assumptions; plain missing-label IPW is insufficient.
• Propensity misspecification: missing important selection variables or poorly calibrated π̂ biases IPW. Check calibration by risk segment, time, policy version and logged selection probabilities. A high propensity-prediction AUC is not proof of correct probability calibration.
• Outcome-regression errors: an overfit m̂ can bias or destabilize finite-sample DR. Use time-respecting out-of-fold or cross-fitted nuisance predictions and keep the evaluated model fixed to avoid accidentally training the evaluand using evaluation labels.
• Extreme weights: π=.01 yields weight 100. Inspect maximum weights, quantiles and ESS; clipping or truncation trades potentially lower variance for bias. Report a sensitivity curve across clipping thresholds, not one silently clipped score.
• Small samples and uncertainty: bootstrap or influence-function standard errors must account for weighting, nuisance-model training, clustered customers, time dependence and any estimation of π. Do not interpret point estimates without intervals and effective sample size.
• DR finite-sample bounds: basic AIPW for a bounded quantity can occasionally produce a value outside [0,1]. Such an estimate flags finite-sample instability or model issues; blindly clipping conceals its sampling behavior.

Production evaluation procedure

• Define the estimand: accuracy, error cost, recall, or mean loss over which decisions, time period, model version and actual/relevant policy?
• Log complete decision cohorts: prediction ID, feature state and timestamp, model/threshold version, selected review/action, policy assignment probability, label-observation status, finalized result and revision history.
• Estimate or recover observation propensities: use known randomized audit probabilities where possible; otherwise model observation R from variables recorded for both labeled and unlabeled decisions. Audit positivity and drift in the labeling policy.
• Build an outcome model using observed outcomes: train with temporal separation or cross-fitting. Check subgroup calibration and predictive validity without target leakage.
• Report alongside each estimate: raw observed fraction, mature/eligible cohort definition, naïve score, IPW (HT or normalized variant clearly stated), DR score, ESS, CIs, propensity diagnostics and subgroup breakdowns.
• Stress-test selection assumptions: alter propensity bounds, vary clipping and models, simulate unmeasured selection and contrast against a genuinely random adjudicated audit sample if available.
• Escalate carefully: if estimates disagree or CI widths are large, investigate labeling coverage and selection policy before automatically retraining or replacing a deployed model.

Engineering trade-offs and what DR does not solve

• Weight-only evaluation is easy to implement but vulnerable to extreme weights and incorrect missingness models.
• DR adds an outcome model and cross-fitting complexity, but can improve resilience to one nuisance model's misspecification under the core missingness assumptions.
• Neither approach makes impossible counterfactual outcomes observable, cures missing-not-at-random feedback by magic, or guarantees accurate small-sample inference.
• This lesson specializes the selective-feedback correction suggested in Question 245. It does not repeat Question 244 on temporal feature leakage, Question 243 on walk-forward backtesting or Question 242 on prequential score aggregation.

Key takeaway: Label-observation bias should be quantified, not ignored. IPW corrects observation-rate imbalance using propensity weights; doubly robust estimation adds a model of outcomes and a residual correction. Both require clear targets, enough overlap, credible logged selection information, and honest uncertainty.

References

• Bang and Robins (2005), Doubly Robust Estimation in Missing Data and Causal Inference Models
• Missing Data: Foundations and Methods, estimating equations and AIPW
• Augmented Inverse Probability Weighting and the Double Robustness Property
• The Selective Labels Problem, KDD 2017