How can rolling-window and prequential evaluation measure model performance under concept drift?

Rolling-window evaluation measures model performance on only the most recent labeled predictions, while cumulative (expanding-window) evaluation averages over all historical predictions. Prequential evaluation (“test-then-train”) defines the *order of evaluation*: predict on each incoming example before its label is used to update the model. Combining prequential scoring with a rolling metric is especially helpful when recent performance changes are obscured by a long history of good predictions.

Why cumulative accuracy can conceal drift

Imagine an online support classifier that makes 12 sequential predictions. The stream's correctness values are: [1,1,1,1,1,1,1,1,0,0,0,0], where 1=correct, 0=incorrect. This is a toy stream intended only to show evaluation mathematics.

Step 1 — Expanding/cumulative evaluation

• After prediction 8, 8/8 = 100% accurate.
• At prediction 10, 8/10 = 80% accurate.
• At prediction 12, 8/12 ≈ 66.67% accurate.
• The final number blends an excellent old regime with poor recent results. If the historical history were 10,000 correct predictions, a few new failures would move the cumulative rate very little.

Step 2 — Rolling accuracy with window W=4

• At prediction 8, the last four outcomes are [1,1,1,1], giving 4/4=100%.
• At prediction 9, the last four are [1,1,1,0] → 3/4=75%.
• At prediction 10, the last four are [1,1,0,0] → 2/4=50%.
• At prediction 11, the last four are [1,0,0,0] → 1/4=25%.
• At prediction 12, the last four are [0,0,0,0] → 0/4=0%.
• Contrast at t=12: cumulative 66.67%, rolling-4 0%. The recent deterioration is more obvious in the short window.

Step 3 — Larger window W=8

• At t=12, the last eight observations comprise four correct and four incorrect predictions → 4/8=50%.
• W=8 varies less due to one individual event, but reacts less sharply to the recent change than W=4. With just four outcomes in a window, each result changes accuracy by 25 percentage points, so small windows can be noisy.

What exactly is prequential (test-then-train) evaluation?

• At event t, receive feature vector X_t and obtain prediction ŷ_t using the model state available before label y_t is known.
• Log prediction ID, timestamp, model version, predicted label or probability, and the decision.
• Only after the true label becomes available, compare ŷ_t with y_t, update relevant evaluation statistics, then allow the model to learn from that labeled example according to the chosen update policy.
• If labels are immediately available, this is the simple loop predict → score → learn → next example. If labels are delayed, predictions can continue before earlier outcomes are scored; each recorded prediction must still be evaluated against the model version that actually generated it.
• Prequential evaluation does not require the model to adapt after every observation: a fixed production model can be evaluated sequentially without training; an online-learning experiment adds the learn step after scoring.
• Crucial distinction: rolling versus cumulative refers to how historical scores are aggregated; prequential refers to when a prediction is made and evaluated relative to training. They can be combined in several ways.

Small test-then-train illustration with actual labels

• Suppose at t=9 a model predicts “billing” for a ticket, but once verified its true class is “technical”. First record the t=9 mistake in the evaluation stream; only then may an online learner update using the t=9 label.
• At t=10, the updated model (if the label was already available) predicts another ticket with its new state. Evaluate that prediction using its future label, not by rerunning the old ticket through an already-updated model.
• Scoring a case *after* learning its outcome would leak target information into its own evaluation and inflate the reported accuracy.

Expanding, sliding, and fading evaluations

• Expanding / landmark window: retain all scores since a fixed beginning. Provides a stable long-run summary but slow response to new drift.
• Sliding/rolling window: retain only the latest W eligible labeled predictions. Captures recent performance better but adds sampling variability; windows can be defined by count or event time, and those definitions differ with variable traffic.
• Fading / exponentially weighted evaluation: place progressively less weight on old outcomes without a hard cutoff. Offers a smooth recency preference but depends on the chosen decay factor. This is an evaluation aggregation choice, not a repetition of Question 239's statistical EWMA alarm chart.
• Recommendation: report both a long-run metric and a recent metric, together with their denominators and uncertainty. Different views answer different questions.

Python example: cumulative versus rolling prequential scores

PYTHON
from collections import deque

# Outcomes generated from predictions made before each label is learned.
correct = [1]*8 + [0]*4
recent4, recent8 = deque(maxlen=4), deque(maxlen=8)
all_correct = 0

for t, outcome in enumerate(correct, start=1):
    # Evaluate frozen prediction first; in a real stream this
    # outcome becomes available only when its label arrives.
    all_correct += outcome
    recent4.append(outcome)
    recent8.append(outcome)
    cumulative = all_correct / t
    r4 = sum(recent4) / len(recent4)
    r8 = sum(recent8) / len(recent8)
    if t in (8, 10, 12):
        print(t, round(cumulative, 4),
              round(r4, 4), round(r8, 4))
    # An optional online model update would occur only now,
    # after the current example was scored.
# t=8:  1.0, 1.0, 1.0
# t=10: 0.8, 0.5, 0.75
# t=12: 0.6667, 0.0, 0.5

Choosing window size and forgetting old results

• Small W: more responsive to abrupt drift, but more susceptible to fluctuations, outliers, uneven traffic, and high-alert volume.
• Large W: steadier estimates and more minority-class examples, but may obscure sudden changes. Select based on the minimum effect size worth detecting, traffic volume, feedback delay and acceptable noise.
• Use a minimum labeled sample count before claiming stability or degradation. If an entire segment has only two positive outcomes in the window, recall will be extremely uncertain.
• Evaluate multiple recent horizons (for example, hourly and weekly) if they match operational response times. Avoid cherry-picking whichever window makes a result look worst.
• Forgetting evaluation results differs from forgetting training data. A rolling accuracy monitor can use W=1,000 while the online model still learns from all earlier data. Sliding-window *training* is a separate model-adaptation design.

Temporal validation and data leakage

• For offline development, split by time: earlier events for model fitting, later events for validation, and a separate subsequent interval for final testing. Use rolling-origin or forward-chaining evaluation rather than randomly mixing future with past.
• Apply a gap/purge when features or label horizons overlap the boundary, and split by entity when repeated customers or near duplicates would leak information.
• Fit preprocessing, encoders, scalers, thresholds and feature selection on training-only history. A timestamp column should never expose future outcomes.
• Keep the recorded historic prediction when assessing a deployed system. Re-evaluating old observations using a model trained later answers a different question and overstates what was actually known at prediction time.

Delayed, incomplete and selective labels

• When ground truth arrives days or weeks later, preserve an event-time prediction ledger and join eventual results by stable identifiers.
• Do not treat pending labels as correct or negative. A rolling metric over *only already observed labels* can be biased toward cases with faster feedback; report label coverage, maturation and delays.
• Compare matured cohorts (for example, predictions old enough for 14-day outcomes) or explicitly adjust for known sampling/selection mechanisms. A time series by label-arrival date may shift for operational reasons even if actual model behavior did not.
• In an online learner, avoid training on labels that were not available yet at the intended model update time. In replay, reproduce the historical label-availability schedule.

Imbalanced streams and model monitoring

• Accuracy can be misleading: if 99% of outcomes are legitimate transactions, an all-legitimate classifier reaches 99% accuracy yet misses every fraud case.
• Alongside recent and cumulative accuracy, track positive-class recall, precision, F1, balanced accuracy, calibration, log loss or business-weighted costs as suitable for the task.
• Use enough positives and negatives for stable windowed rates and AUC, and report sample volumes. For very rare positive labels, count-based windows may be more useful than short fixed-time windows.
• Track separate segments (channels, product types, geographies) because an aggregate rate can conceal drift in a critical group.

Applications, limitations, and production use

• Applications: fraud detection, online recommendation, customer support routing, predictive maintenance and forecasting quality where data arrive chronologically.
• Prequential strengths: reflects the information a system actually had before each prediction, supports incremental updates, and produces a continuous performance record.
• Limitations: short windows are noisy; real drift onset is unknown; feedback may be delayed or selective; retraining and changing model versions complicate comparison.
• A falling rolling score is evidence of recent performance deterioration, not definitive proof of P(Y|X) drift: label-policy changes, pipeline failures and shifting class mix can also affect it.
• When a sustained decline occurs, check data quality and cohort comparability, investigate segment-specific metrics, then evaluate a candidate response on untouched subsequent data. This lesson explains how to measure the change, not the business approval process for retraining (Question 241).

Key takeaway: Use prequential ordering to prevent evaluation leakage, and rolling-window aggregation to see recent degradation that cumulative accuracy can hide. Interpret both with time-respecting labels, window sizes and class-aware metrics.

References

• Data stream mining: methods and challenges for handling concept drift
• Experimenting with prequential variations for data stream learning evaluation
• Delayed labelling evaluation for data streams
• scikit-learn: TimeSeriesSplit
• Prequential AUC for data streams with concept drift