How can machine learning models be evaluated when ground-truth labels are delayed, missing, or selectively observed?

When labels are delayed, missing, or selectively observed, the set of predictions with known outcomes may not represent the predictions a model actually made. Reliable evaluation requires a prediction-to-label ledger, explicit maturity and coverage rules, and special treatment for selection bias. A reported accuracy among observed labels is not automatically the population accuracy.

Three different label problems

• Delayed: outcome exists or will be determined, but its feedback arrives days or weeks later; e.g. fraud chargebacks or returns.
• Missing or censored: some outcomes never arrive or are unknown at the analysis cutoff. A pending 30-day outcome must not be treated as negative merely because no event has yet been observed.
• Selectively observed: probability of obtaining a label depends on actions, review policy, customer segment, or latent risk. For example, manual investigation may preferentially label suspicious transactions; this is often missing not at random and is not solved by simply waiting.

Beginner-friendly numerical example: naïve versus mature-cohort accuracy

Consider a fraud classifier that made 100 predictions in a recent cohort. At the reporting cutoff:

• Only 40 of 100 predictions have finalized ground-truth labels; 36 of these 40 predictions were correct and 4 wrong.
• Label coverage = 40/100 = 40%.
• Observed-label accuracy = 36/40 = 90%. This describes only the 40 labeled examples, not necessarily all 100.
• Wrong shortcut A — treat all 60 pending outcomes as correct: (36+60)/100 = 96% reported “accuracy.” This is invalid because none of the 60 was actually verified.
• Wrong shortcut B — assume 90% equals true accuracy for all 100: this requires labels to be representative under an appropriate missingness mechanism, which is not established.
• Without assumptions, because 60 outcomes are unknown, total accuracy is bounded by 36/100=36% (all pending ultimately wrong) and (36+60)/100=96% (all pending right). This is a deliberately broad identification interval, not a statistical confidence interval.

Now use an older, sufficiently aged cohort of 100 predictions with the same target definition and a prespecified 30-day label-maturation window:

• 90 outcomes are finalized (label coverage 90%); 72 labeled predictions are correct.
• Mature-cohort observed accuracy = 72/90 = 80%, with 10% still unresolved; this is much more mature but not automatically unbiased.
• If those remaining 10 are not representative, even 80% could misrepresent full-cohort accuracy; population accuracy is between 72% and 82% without assumptions about the 10.
• The apparent difference between recent 90% observed and older 80% mature observed must not be interpreted as a real 10-point improvement: the cohorts and observation mechanisms differ.
• Compare similarly matured cohorts at the same age, with comparable populations, outcome rules, minimum coverage and uncertainty estimates before asserting a performance trend.

Event time, label arrival, and a reliable prediction-to-label join

• Store immutable prediction_id, entity_id, prediction_event_time, model_version, score/probability, predicted decision, label_definition_version, and the relevant action or exposure.
• Store a separate label record with prediction_id, outcome_event_time, label_first_observed_at, label_finalized_at, revision_timestamp, label_value and data-source provenance.
• Join on a stable prediction/decision identifier, not just the customer ID or the latest label. A customer can have many distinct transactions and revised outcomes.
• At a fixed report cutoff T, a label is eligible only if label_finalized_at ≤ T and it belongs to the target definition and prediction horizon; never use a label received after T in an as-of-T backtest.
• Prediction event time governs which cohort a decision belongs to; label arrival time governs when its result may be evaluated. Report both views explicitly to prevent interpretation errors.
• Version corrections and late appeals rather than overwriting the entire history, so earlier monitoring reports can be reproduced.

Defining a maturity window

• For a 30-day return/fraud target, a prediction made yesterday does not have a complete 30-day outcome even if no return or chargeback has appeared. Define a decision horizon and, when necessary, additional administrative lag.
• If reliable finalization requires 30 days of outcome exposure plus 7 days of ingestion/adjudication lag, an illustrative cohort may be considered eligible for reporting only 37 days after its decision time. This is a policy example, not a universal definition.
• Track label coverage conditional on cohort age, e.g. fractions finalized by day 7, 14, 30 and 37. Labels can arrive at very different rates for different classes or regions.
• Censoring: cases with insufficient follow-up are right-censored, not known non-events. For time-to-event tasks, survival methods such as Kaplan–Meier, IPCW or other censoring-aware estimators may be more suitable than binary accuracy, under their stated assumptions.
• Keep cohorts separately for immature, matured-but-unresolved, revised and fully finalized labels. Different reporting cutoffs produce different estimates.

Why selective labels can distort performance

Suppose the system's action determines which customer is investigated or which loan is approved, so some outcomes are never observed for rejected or uninvestigated cases.

• The labeled sample can overrepresent high-risk transactions, difficult cases, or customers reviewed by human operators.
• Coverage ≠ representativeness. Even 95% overall coverage can hide complete absence of one critical small subgroup.
• Stratify by risk band, class, region, model score, human review route and decision. Monitor who is missing, not only how many labels are missing.
• A change in investigation policy can change measured precision/recall or observed loss without any actual change in model quality.

When inverse-propensity weighting (IPW) can help

If the probability πᵢ=P(label observed | information available for case i) is known or credibly estimated and positive, observed examples can receive weight 1/πᵢ to compensate for unequal observation rates. It requires conditional ignorability / missing at random given recorded variables, positivity (πᵢ>0), correct policy/propensity estimation, and sufficient overlap. It cannot recover groups that are *never* labeled or correct for unmeasured factors that determine both labeling and outcome.

Toy IPW numerical demonstration with 100 predictions:

• High-risk stratum: 20 total cases; probability of observing a label π=0.50, so 10 labels are observed; 6 of these are correct.
• Low-risk stratum: 80 total cases; observation probability π=0.25, so 20 labels are observed; 18 of these are correct.
• Naïve observed-only accuracy (6+18)/(10+20)=24/30=80%.
• Apply inverse-probability weights: high-risk weights 1/0.50=2; low-risk weights 1/0.25=4.
• Estimated correct-count total 6×2+18×4=12+72=84; estimated cohort count 10×2+20×4=20+80=100.
• Self-normalized IPW accuracy = 84/100 = 84%. The 4-point increase is an illustrative correction for overrepresented high-risk observations, not the guaranteed true accuracy.
• Random sampling variability and estimated propensities create uncertainty. Very low π leads to unstable large weights: report effective sample size, propensity distribution, uncertainty, and sensitivity to weight clipping.
• This is valid only when outcomes are well-defined for the unobserved cases and selection is ignorable after conditioning. In systems where the policy action changes the potential outcome itself, ordinary IPW of observed labels is not enough without a correctly specified causal estimand and assumptions.

Simple, reproducible Python example

PYTHON
recent_total, recent_labeled, recent_correct = 100, 40, 36
print("coverage:", recent_labeled / recent_total)         # 0.40
print("observed accuracy:", recent_correct / recent_labeled) # 0.90
print("invalid pending=correct:", (36 + 60) / 100)        # 0.96

older_total, older_labeled, older_correct = 100, 90, 72
print("mature coverage:", older_labeled / older_total)     # 0.90
print("mature observed accuracy:", older_correct/older_labeled) # 0.80

naive = (6 + 18) / (10 + 20)
ipw = (6 / .50 + 18 / .25) / (10 / .50 + 20 / .25)
print("observed:", naive, "weighted:", ipw)              # 0.80, 0.84

Proxy metrics while waiting for ground truth

• Monitor score distributions, feature availability, prediction entropy, confidence calibration proxies, routing changes, approval rates, system latency and review queue health. These can be useful early-warning indicators but are not replacements for validated label-dependent error estimates.
• Use earlier signals whose relationship to the final target is stable and empirically verified. Beware proxy drift and changes to manual-review selection.
• If labels are very slow, consider a small, representative adjudicated sample or authorized, appropriately designed auditing; do not assume randomization is feasible or ethical in high-stakes decisions.

Temporal monitoring, uncertainty and production operations

• Publish metric as-of timestamp, cohort event-time interval, target horizon, label coverage, maturity age distribution, sample count, class balance, model versions and uncertainty bands.
• Report separate early provisional and matured final performance. Compare like-aged cohorts to avoid artificial deterioration or improvement caused by feedback latency.
• Estimate confidence intervals using appropriate methods for weighted metrics or clustered/repeated-entity cases; a naïve binomial interval may be too narrow under selection or dependence.
• For rare classes, emphasize class-specific recall, precision, cost-weighted loss and appropriate uncertainty; accuracy can conceal missed fraud.
• Audit changes in human-review policies, labeling budgets, downstream business rules and outcome definitions. Maintain backfills with versioned as-of reports.
• Alerting on untrusted or partially observed loss can amplify error: delay strong actions until evidence is adequate, while allowing investigation on unlabeled proxies.

Common failures and limitations

• Treating pending labels as negative/correct creates optimism and class-dependent bias.
• Only reporting labeled-case accuracy suppresses unresolved cases and can move merely because the review queue changed.
• Assuming long age means fully labeled: some labels never arrive or are systematically censored.
• Assuming IPW fixes everything: unknown propensities, zero observation probabilities, unobserved confounders, action-dependent outcomes and extreme weights can defeat identification or precision.
• A confident interval is not always a trustworthy estimate: statistical precision does not fix a biased labeling mechanism.
• High implementation costs: immutable prediction ledgers, revision history, adjudication, cohort reporting and robust inference demand storage and operational resources.

Relationship with nearby ML lessons

• Question 244: prevents future information entering historical features and training labels. This lesson asks what performance can be estimated *when some valid target labels are not yet observable at all*.
• Question 243: defines offline chronological training/backtesting folds. This lesson defines label observability, censoring and comparability within those folds.
• Question 242: teaches test-then-train order and online metric windows. This lesson extends evaluation to nonuniform, missing and policy-selected feedback, including cases never resolved.
• This is distinct from the individual drift detectors (Q234–239) and their benchmark or retraining rules (Q240–241).

Key takeaway: Never convert unknown into negative or correct. Use versioned prediction-to-label joins, comparable matured cohorts, honest coverage and uncertainty, and explicit selection assumptions before trusting observed performance.

References

• Grzenda, Gomes & Bifet, Delayed labelling evaluation for data streams (2020)
• Lakkaraju et al., The Selective Labels Problem (KDD 2017)
• Label Delay in Online Continual Learning, NeurIPS 2024
• Handling delayed labels in temporally evolving data streams (IEEE Big Data 2016)