How does CUSUM (Cumulative Sum Control Chart) detect small, persistent changes in a streaming model's error or loss?

CUSUM (Cumulative Sum Control Chart) is a sequential change detector that accumulates small, persistent deviations from a stable reference mean. A single moderately unusual observation may produce no alarm, while several observations consistently above or below the expected level build enough evidence to signal a possible mean shift. In machine learning, the monitored signal can be per-example loss, aggregated error rate, or another appropriately scaled performance metric.

The two-sided tabular CUSUM

• Choose a reference mean μ₀ from a trusted, stable baseline. Keep it fixed while monitoring; recomputing it on drifting data can conceal the shift.
• Define k ≥ 0, the reference or allowance value, which discounts small departures. In the usual mean-shift design, k ≈ Δ/2, where Δ is the shift in the monitored signal's units that you want to detect promptly.
• Define h > 0, the decision threshold, expressed in the same accumulated-signal units as the CUSUM.
• Initialize C₀⁺ = C₀⁻ = 0.
• For each new observation xₜ, compute Cₜ⁺ = max(0, Cₜ₋₁⁺ + xₜ − μ₀ − k). This accumulates evidence of an increase.
• Also compute Cₜ⁻ = max(0, Cₜ₋₁⁻ + μ₀ − xₜ − k). This accumulates evidence of a decrease, using a nonnegative lower-side statistic.
• Signal an alarm if Cₜ⁺ > h or Cₜ⁻ > h. An implementation may define equality as alarming; document that convention.
• The max-with-zero operation automatically clears accumulated evidence when newer observations no longer support that direction. A warning state is not intrinsic to this basic two-sided chart.

Beginner-friendly numerical example: a ticket classifier's monitored loss

Suppose a classifier's validated baseline mean loss is μ₀ = 0.20. A small increase or decrease of 0.10 is worth noticing, so choose k = 0.10/2 = 0.05. For the teaching example, set h = 0.30. Monitor the next eight per-observation numeric losses: [0.20, 0.10, 0.22, 0.20, 0.32, 0.34, 0.33, 0.35].

Step-by-step calculations:

• t=1, x=0.20: C⁺=max(0,0+0.20−0.20−0.05)=0.00; C⁻=max(0,0+0.20−0.20−0.05)=0.00.
• t=2, x=0.10: C⁺=max(0,0+0.10−0.20−0.05)=0.00; C⁻=max(0,0+0.20−0.10−0.05)=0.05. This isolated low loss does not cross h.
• t=3, x=0.22: C⁺=max(0,0+0.22−0.20−0.05)=0.00; C⁻=max(0,0.05+0.20−0.22−0.05)=0.00. The lower-side evidence resets.
• t=4, x=0.20: both statistics remain 0.00.
• t=5, x=0.32: C⁺=max(0,0+0.32−0.20−0.05)=0.07; C⁻=0.00.
• t=6, x=0.34: C⁺=0.07+0.34−0.20−0.05=0.16; C⁻=0.00.
• t=7, x=0.33: C⁺=0.16+0.33−0.20−0.05=0.24; C⁻=0.00.
• t=8, x=0.35: C⁺=0.24+0.35−0.20−0.05=0.34; C⁻=0.00.
• Alarm at t=8: C⁺ = 0.34 > h = 0.30. Repeated elevated losses accumulated enough evidence despite each observation being only modestly above the baseline.

Why a single spike need not trigger the same way: A lone observation of 0.32 starting from C⁺=0 produces 0.07, far below h. Several consecutive moderately elevated observations add up. A sufficiently large isolated spike *can* still cross the threshold, so CUSUM is not immune to outliers.

Reproduce the calculations in Python

PYTHON
values = [0.20, 0.10, 0.22, 0.20, 0.32, 0.34, 0.33, 0.35]
mu0, k, h = 0.20, 0.05, 0.30
upper = lower = 0.0

for t, x in enumerate(values, start=1):
    upper = max(0.0, upper + x - mu0 - k)
    lower = max(0.0, lower + mu0 - x - k)
    alarm = upper > h or lower > h
    print(t, round(upper, 2), round(lower, 2), alarm)
    if alarm:
        print("First CUSUM alarm at", t)  # 8
        break

Choosing the baseline, k and h

• Baseline μ₀: estimate from a stable reference window after validating model version, data quality, label completeness, feature definitions and normal seasonality. Use consistent loss units.
• Allowance k: larger k disregards more small deviations, potentially missing slow shifts; smaller k is sensitive to smaller sustained changes but can accumulate random fluctuations.
• Threshold h: larger h reduces nuisance alarms but can lengthen detection delay; smaller h reacts faster at a higher false-alarm risk.
• Standardization: when monitoring standardized process measurements, SPC texts often recommend k near half the target shift in standard deviation units and h around 4–5 standard deviations as a starting design. Our h=0.30 is a simple numerical illustration, not a calibrated production threshold.
• Calibrate average run length: evaluate how many observations are expected between false alarms under stable conditions (in-control ARL) and how quickly shifts of practical interest are found (out-of-control ARL). Use held-out stable and simulated drift streams.
• Reference uncertainty and repeated monitoring: if baseline mean or variance is poorly estimated, nominal false-alarm behavior may not hold. Avoid continually retuning parameters on the same evaluation stream.

Error and loss signals; delayed labels

• For classification, use per-example 0/1 error, log loss, or an appropriate cost-sensitive loss. The example uses real-valued loss, so its values are not interpreted as 0/1 error indicators.
• Aggregate batch errors can be monitored, but changing batch sizes lead to different measurement variance; standardize or use a chart designed for counts/proportions.
• If the metric requires ground-truth labels, join labels to prediction IDs and model versions, account for label delay and selective feedback, and choose event-time or arrival-time ordering deliberately.
• Never assume unanswered cases are correct. Fraud feedback may be weeks late; monitor a valid provisional signal separately if needed.

CUSUM versus Page–Hinkley and ADWIN

• Page–Hinkley (Question 237): accumulates deviations from a running mean, tracks the historical cumulative minimum, and alarms when the resulting excursion exceeds λ. Basic CUSUM instead uses a predefined in-control reference μ₀ and explicitly accumulates upper and lower departures with allowance k.
• ADWIN (Question 236): uses statistically evaluated differences between subwindow means and adapts the window length. CUSUM does not inherently maintain a variable window or choose a data-dependent cut.
• DDM/EDDM (Questions 234–235): specialize in binary mistake rates or distances between mistakes; generic tabular CUSUM can monitor numeric losses and other process measurements.
• None is universally superior. Compare delayed detection, false alarms, memory/compute budgets, label availability and sensitivity to specific shift sizes.

Resetting and production response

• A zero-valued upper or lower CUSUM is a local automatic reset for that direction; this is different from resetting the whole detector after an alarm.
• On alarm, identify potential pipeline errors, label-definition changes, altered traffic mix, model deployment changes, or genuine deterioration. A mean shift in monitored loss is not by itself proof of causal concept drift.
• If a model is legitimately replaced or the process is repaired, establish a new trustworthy baseline and deliberately reset or reinitialize both sums. Avoid automatically resetting and repeating an alarm indefinitely on unchanged faulty input.
• Keep model-versioned metrics, feature cohorts, alert histories and rollback procedures. Check class-specific performance because aggregate mean loss can conceal damage in minority groups.

Assumptions and limitations

• Standard statistical-control guarantees rely on an adequately stable in-control process and assumptions about the noise distribution and dependence. Strong temporal autocorrelation, clustered events and heavy-tailed losses can cause spurious alarms.
• CUSUM is most directly sensitive to changes in the mean of the monitored signal; other changes (for example variance-only drift) may be missed or require additional monitoring.
• In a live ML service, significant input composition changes or mislabeled events can increase measured loss without changing the fundamental target relationship.
• Production charts often need explicit outlier handling, seasonality adjustment, subgroup diagnostics, multiple-monitor alert management and threshold calibration.

Key takeaway: Two-sided CUSUM converts persistent small mean deviations into positive or negative accumulated evidence. It is a lightweight alarm for a changing performance signal, but deployment decisions require further root-cause and performance validation.

References

• NIST: CUSUM control charts and tabular recursions
• NIST: CUSUM average run length
• NIST Dataplot: CUSUM control chart