How can multiple hypothesis testing and false discovery rate control improve production machine learning monitoring across many metrics and segments?

Multiple testing matters whenever a production ML dashboard checks performance across many features, classes, regions, model versions, or user groups. Even if each individual statistical test has a small false-positive rate, alerting on every unadjusted p-value can generate an unacceptable number of false alarms. Bonferroni controls the probability of at least one false rejection in a predefined family; Benjamini–Hochberg (BH) controls the expected fraction of false discoveries among rejected hypotheses, under its dependence assumptions.

1. Begin with the monitoring question, not a p-value

• Specify the null hypothesis for each monitored comparison, e.g., “fraud recall in the current mature cohort equals the approved baseline” or “the deployment did not change the mean loss in this region.” Use a one-sided hypothesis for deterioration if chosen before looking at the data.
• Define the family of tests before checking which p-values are small. For example, six prespecified performance slices for one model, one time window and one metric are a family of six.
• A valid p-value under its null has P(p ≤ u) ≤ u for every threshold 0≤u≤1. Good multiplicity control depends on valid underlying p-values, including correct handling of temporal dependence, selection, delayed outcomes and repeated users.
• Specify the operational false-alarm tolerance, cost of missed degradation, required effect size and review capacity; statistical significance alone does not establish business importance.

2. Difference between FWER and FDR

• Family-wise error rate (FWER) is P(V≥1), the probability of making *at least one* false rejection among all hypotheses in the family. It is stringent and useful for high-consequence automatic interventions.
• False discovery rate (FDR) is E[V/max(R,1)], where V is the number of false rejections and R is the total number of rejections. If R=0, the false-discovery proportion is defined as zero.
• At q=0.05, FDR control means the expected fraction of false discoveries across repeated, similarly designed experiments is at most 5% under the method's assumptions. It does not mean each alert has a 5% chance of being wrong, that exactly 5% of today's alerts are false, or that no false alert can occur.
• For illustration, six *independent* true-null tests at unadjusted α=.05 would yield P(at least one false alarm)=1−.95⁶≈26.49%. With correlated dashboard tests this exact probability changes, but unadjusted per-test control still does not ensure global FWER.
• Bonferroni is conservative but robust to arbitrary dependence among valid p-values. BH generally has greater power when many genuine changes exist, but its standard guarantee requires independent p-values or suitable positive regression dependence (PRDS) on the true-null subset; not all correlation patterns qualify.

3. Beginner-friendly numerical example: six model-monitoring segments

Suppose a fraud model's verified loss or recall is tested in six *prespecified* monitoring slices during one *fixed, completed evaluation window*, using valid comparison tests. These are illustrative p-values, not measurements from a real system:

• Segment A, merchant channel: p=0.001
• Segment B, international payments: p=0.012
• Segment C, mobile payments: p=0.021
• Segment D, new users: p=0.030
• Segment E, returning users: p=0.200
• Segment F, other traffic: p=0.700

The p-values are already sorted from smallest to largest. Choose overall 0.05.

Step 1 — Unadjusted tests

• Comparing each p-value directly with 0.05 would mark A, B, C, and D, but this lacks a family-wise 5% guarantee because six opportunities were tested.

Step 2 — Bonferroni correction

• With m=6 hypotheses and target FWER α=.05, the per-test cutoff is α/m=.05/6≈.008333.
• A (.001) passes; B (.012), C (.021), D (.030), E (.200), and F (.700) do not.
• Bonferroni-adjusted p-values min(1,6p) are [.006,.072,.126,.180,1,1].
• Decision: only A is significant at family-wide 5%. Bonferroni needs no independence assumption for controlling FWER, given valid individual p-values and a properly defined family.

Step 3 — Benjamini–Hochberg (BH) procedure

• Sort all m=6 p-values in ascending order; with target FDR q=.05, compare rank i against (i/m)q.
• Rank 1 (A): .001 vs .008333 → passes.
• Rank 2 (B): .012 vs .016667 → passes.
• Rank 3 (C): .021 vs .025000 → passes.
• Rank 4 (D): .030 vs .033333 → passes.
• Rank 5 (E): .200 vs .041667 → fails.
• Rank 6 (F): .700 vs .050000 → fails.
• Select the largest passing rank k=4 and reject all hypotheses at ranks 1 through 4. This is the step-up rule: do not simply reject only individually passing ranks when there are irregular p-values.
• Decision: A, B, C, D are discoveries under BH at FDR 5%, assuming its applicable dependence conditions.
• The BH adjusted p-values, from monotone reverse cumulative minima of p_(i)m/i, are approximately [.006,.036,.042,.045,.240,.700].
• Thus Bonferroni flags 1 of 6; BH flags 4 of 6. This shows the difference between a stricter guarantee against *any* false alarm and accepting control of the expected false-discovery proportion, not proof that the four BH alerts are truly valid.

4. Reproduce the exact adjustment decisions in Python

PYTHON
import numpy as np
from statsmodels.stats.multitest import multipletests

names = ["A", "B", "C", "D", "E", "F"]
pvals = np.array([.001, .012, .021, .030, .200, .700])
alpha = .05

bonf_reject, bonf_adj, *_ = multipletests(
    pvals, alpha=alpha, method="bonferroni")
bh_reject, bh_adj, *_ = multipletests(
    pvals, alpha=alpha, method="fdr_bh")

for name, p, b, f, pb, pf in zip(
    names, pvals, bonf_reject, bh_reject, bonf_adj, bh_adj
):
    print(name, p, bool(b), bool(f), round(pb, 3), round(pf, 3))
# A .001 True  True  .006 .006
# B .012 False True  .072 .036
# C .021 False True  .126 .042
# D .030 False True  .180 .045
# E .200 False False 1.000 .240
# F .700 False False 1.000 .700

If a monitoring team chooses thresholds or tests after seeing their results, the original guarantees may no longer apply. Review the code and statistical validity of each underlying p-value, not only the correction procedure.

5. Dependence assumptions and overlapping monitoring segments

• Metrics from the same examples can be correlated, e.g., fraud recall, error rate and log loss. Regions and merchant cohorts can overlap. Multiple models evaluated on the same cohort produce paired, dependent comparisons.
• Bonferroni's union bound controls FWER for valid p-values without assuming independence, although it may sacrifice statistical power.
• BH is justified under independence and certain positive-dependence structures (PRDS); arbitrary overlap does not automatically satisfy that condition. If assumptions are unclear, consider a validated alternative such as Benjamini–Yekutieli (BY) for general dependence or an appropriately designed resampling-based procedure.
• BY uses a more conservative adjustment involving H_m=Σ(i=1…m) 1/i; for m=6 this harmonic factor is 2.45. It can be much less powerful.
• Holm's step-down procedure also controls FWER under arbitrary dependence and can be preferable to plain Bonferroni for confirmatory, high-stakes use.

6. Repeated daily dashboards and sequential testing

• BH on six fixed-window p-values protects FDR only for the *specified six-test batch* under its assumptions. Running a fresh six-test batch each hour forever does not automatically control a global lifetime error budget.
• Question 248 explains optional stopping and confidence sequences for a *single continuously monitored stream*. This lesson explains multiplicity across many simultaneous or selected streams. Both issues can be present, so they must be handled together.
• Use valid anytime/sequential p-values or e-values when continuous peeking is required, plus an across-stream correction or a rigorously designed online FDR algorithm (e.g., LORD or SAFFRON) when hypotheses arrive sequentially.
• A fixed family-size Bonferroni allocation α/m is not sufficient if the same family is re-tested indefinitely without spending or maintaining a time-valid guarantee.
• Track policy-defined alert episodes, cooldown periods and model-version resets so the same incident is not counted as dozens of independent discoveries.

7. Segment selection, multiple versions, and statistical power

• Selection after looking at the data—such as displaying only the five worst of 100 segments and adjusting as if only five were tested—invalidates the stated family. Include the full search space or use a justified selective-inference or prespecified screening design.
• Model comparisons: testing one model against five challengers, repeated after many hyperparameter searches, adds multiple comparisons. Keep a truly held-out final evaluation and pair outcomes where appropriate.
• Low volume: rare-class recall over seven finalized positives is noisy. Correcting p-values does not fix tiny sample sizes, dependence, target leakage or a biased label-observation policy.
• More stringent FWER control often lowers power and may delay detection of true changes; FDR control generally yields more useful discoveries in broad exploratory monitoring but allows some false discoveries in expectation.
• Predefine practical effect-size thresholds, minimum label counts, confidence criteria, and operational risk before alerting on p-values alone.

8. Production alert calibration and operations

• Define families by business question and analysis scope: prespecified metric × model version × cohort × evaluation time. Document which comparisons are confirmatory or exploratory.
• Use one-sided or two-sided valid statistical tests matched to the defined null, with correct denominators, class balance and treatment of repeat entities.
• Attach to each alert its raw p, adjusted p or BH rejection status, effect size, uncertainty interval, sample volume, label maturity, segment definition and family size.
• Route FWER-controlled signals to strict incident workflows where one false alarm may trigger an expensive or harmful intervention. Use FDR-controlled candidates for prioritizing human investigation when false positives can be reviewed cheaply.
• Quantify business costs: false-alert investigation, missed critical regressions, label acquisition, customer harm, rollout/rollback cost, and staff alert fatigue. Multiplicity correction affects statistical error, not operational severity.
• Validate thresholding retrospectively on representative stable and change periods, with seasonal, segment-mix and delayed-feedback effects. Monitor false-alert rate, detection delay and measured benefit of responses.
• A rejected null suggests a statistical difference under the model assumptions. It does not by itself establish causal concept drift, adequate effect size, or that production retraining should happen automatically.

9. Common mistakes and limitations

• Calling FDR the “probability today's alert is false” instead of an expectation across rejected sets.
• Treating six selected p-values as the entire family when hundreds of metrics were screened.
• Applying BH to arbitrary highly dependent tests without checking its conditions, or assuming Bonferroni requires independent tests.
• Repeatedly applying fixed-window tests to streaming data and ignoring optional stopping.
• Combining multiple model comparisons, post-hoc segment definitions and tuning without counting their selection opportunities.
• Overreacting to tiny p-values with trivial effects or assuming nonsignificance proves no degradation.
• Treating unavailable labels as negative or correct; selective observation invalidates raw tests before multiplicity correction can help.
• Failing to specify if error control applies to a single batch, repeated analyses or the complete continuous monitoring program.

Relationship to neighboring lessons

• Question 247: fixed-sample CIs around individual metrics; this lesson manages many simultaneous statistical conclusions.
• Question 248: repeated looks, optional stopping and time-uniform inference for a monitored stream; this lesson handles comparisons *across metrics and segments* and notes how to combine temporal and cross-stream control.
• Questions 234–239: particular drift detectors. This lesson selects the interpretation and error budget for multiple alerts without repeating detector formulas.
• Questions 245–246: label observation and selection bias. Multiplicity corrections do not repair invalid raw p-values from those problems.

Key takeaway: Determine the monitoring family and consequences first. Use Bonferroni/Holm when avoiding even one false alarm matters; use BH/FDR for broader discovery and triage when its assumptions fit. Add sequential validity if monitoring continues through time.

References

• Benjamini & Hochberg (1995), Controlling the False Discovery Rate
• statsmodels: multipletests API
• Benjamini & Yekutieli (2001), FDR under dependency
• Ramdas et al., Online control of the false discovery rate with decaying memory