How can the chi-square test detect categorical feature distribution drift?

The Pearson chi-square test of homogeneity evaluates whether the distribution of a categorical feature differs between a reference dataset and a recent production sample. For drift monitoring, it tests whether categories such as payment method, region, or device type have the same underlying proportions across two periods. Unlike numerical drift methods, it works with category counts, not ordered distances.

Hypothesis and formula

• Null hypothesis (H0): the categorical feature has the same population proportions in the reference and production periods.
• Alternative (H1): at least one category proportion differs.
• Make a 2 × K contingency table: rows for reference and production, columns for the K categories.
• For each cell: expected count E = (row total × column total) / grand total.
• Chi-square statistic: χ² = sum over all cells of (observed − expected)² / expected.
• Degrees of freedom = (number of rows − 1) × (number of columns − 1); for two periods and K categories, df = K − 1.

Step-by-step numerical example: payment-method drift

An ecommerce classifier monitors a customer payment-method feature with three categories: Card, UPI, and Cash.

Observed reference counts (100 orders):

• Card 50, UPI 30, Cash 20.

Observed current counts (100 orders):

• Card 30, UPI 40, Cash 30.

Step 1 — Calculate totals

• Reference row total 100, production row total 100, grand total 200.
• Category totals: Card 80, UPI 70, Cash 50.

Step 2 — Expected frequencies assuming no distribution change

• Reference Card: 100 × 80 / 200 = 40; current Card also 40.
• Reference UPI: 100 × 70 / 200 = 35; current UPI also 35.
• Reference Cash: 100 × 50 / 200 = 25; current Cash also 25.
• Each row's expected counts total 40 + 35 + 25 = 100.

Step 3 — Compute chi-square contributions for all six cells

• Reference Card: (50 − 40)²/40 = 2.5000.
• Current Card: (30 − 40)²/40 = 2.5000.
• Reference UPI: (30 − 35)²/35 = 0.7143.
• Current UPI: (40 − 35)²/35 = 0.7143.
• Reference Cash: (20 − 25)²/25 = 1.0000.
• Current Cash: (30 − 25)²/25 = 1.0000.

Step 4 — Add contributions

• χ² = 2.5 + 2.5 + 0.7143 + 0.7143 + 1 + 1 = 8.4286.

Step 5 — Degrees of freedom and p-value

• df = (2 − 1) × (3 − 1) = 2.
• The asymptotic chi-square distribution gives p ≈ 0.0148.
• At a predeclared significance level of 0.05, reject H0: the data provide evidence that the payment-method distribution has changed.
• This is not a 98.52% probability that drift occurred, and it does not establish that the classifier has become less accurate.

Python verification

PYTHON
import numpy as np
from scipy.stats import chi2_contingency

counts = np.array([
    [50, 30, 20],  # Reference: Card, UPI, Cash
    [30, 40, 30],  # Production
])

result = chi2_contingency(counts, correction=False)
print(round(result.statistic, 4))  # 8.4286
print(round(result.pvalue, 4))     # 0.0148
print(result.dof)                  # 2
print(result.expected_freq)       # [[40., 35., 25.], [40., 35., 25.]]

How large is the change?

• Statistical significance and effect size are not the same. Large samples can make even tiny shifts significant.
• A commonly reported association effect size is Cramér's V = sqrt(χ² / [N × min(R−1, K−1)]), where N is the grand total.
• Here V = sqrt(8.4286 / (200 × 1)) ≈ 0.205. This quantifies association between time period and payment category in the observed data.
• Do not assume one universal meaning of "small" or "large" V; interpret against business tolerance, category makeup, and historical baselines.

Assumptions and small or rare categories

• Observations should represent independent events under the sampling design. Repeated purchases by the same person, overlapping windows, or time dependence can invalidate naive p-values.
• Categories must be mutually exclusive and identically defined in both periods; use consistent handling of missing and newly appearing values.
• The usual chi-square reference distribution is an approximation, so it can be unreliable with small expected counts. The often-used rule of thumb is that expected cell counts should generally be at least 5; exact constraints depend on context.
• For a 2 × 2 table with sparse counts, Fisher's exact test is a standard alternative. For larger sparse tables, investigate suitable exact, Monte Carlo, or permutation methods; do not blindly apply a 2 × 2-only recipe.
• If categories are very rare, combine them only when the grouping is substantively defensible and documented. Do not merge classes merely to obtain the desired p-value.
• For clustered users or strongly dependent observations, consider user-level analysis or appropriate cluster/block resampling.

Applications and production monitoring

• Applications: payment methods, region, browser type, device model, signup channel, document category, and other discrete features.
• Compare aligned reference and recent windows, checking changes in category definitions, missingness, traffic volume and seasonality.
• Monitor contributions by category: here Card contributes 5.0 of the total 8.4286, helping focus investigation.
• Use a stable significance level and correct or otherwise account for multiple comparisons if testing dozens of features each day.
• Pair statistical alerts with effect sizes, absolute proportion shifts, model performance, segment outcomes, and business cost.
• If drift results from a pipeline bug, repair the pipeline. If real behavior changed, validate whether the deployed model is actually degraded before recalibration or retraining.

Limitations

• The test detects changes in marginal categorical frequencies, not joint feature shifts or changes in P(Y|X) (concept drift).
• Large sample sizes can yield very small p-values for negligible effects; small samples can fail to detect important drift.
• It does not convey category ordering or a physical distance between categories.
• A significant result is a signal for diagnosis, not a mandate to retrain or evidence of a causal explanation.

Key takeaway: The chi-square homogeneity test compares observed category counts with counts expected under stable proportions. Report χ², degrees of freedom, p-value, effect size, and the affected categories before deciding whether to act.

References

• SciPy: chi2_contingency
• SciPy: fisher_exact