How do the Kolmogorov–Smirnov (KS) test and Wasserstein distance detect numerical feature drift?

The two-sample Kolmogorov–Smirnov (KS) test and 1D Wasserstein distance compare reference and current numerical feature samples without requiring PSI's fixed bins. They measure different properties: KS captures the largest gap between empirical cumulative distributions, whereas Wasserstein-1 measures the total horizontal displacement of probability mass.

Core definitions

• An empirical cumulative distribution function (ECDF) at value x is the fraction of sample observations ≤ x.
• KS statistic: D = sup_x |F_ref(x) − F_cur(x)|, the maximum absolute ECDF separation. Its value lies in [0,1], is dimensionless, and is unchanged by a strictly increasing transformation of both samples.
• 1D Wasserstein-1 distance: W1 = integral |F_ref(x) − F_cur(x)| dx. It can also be calculated by optimally matching sorted equal-weight observations. W1 has the same units as the original feature.
• Both are univariate distribution-comparison tools, not direct tests of model accuracy.

Beginner-friendly numerical example: monthly income feature

A model historically saw these incomes in ₹ thousands:

• Reference (n=5): [40, 50, 60, 70, 80].
• Current production (n=5): [65, 75, 85, 95, 105].

Both samples are sorted. Each current value is 25 thousand rupees larger than its corresponding reference value.

Step 1 — Compute the ECDFs at informative values

• At x=60: reference has 3/5 observations ≤ 60, so F_ref(60)=0.60; current has 0/5, so F_cur(60)=0.00. The gap is 0.60.
• At x=70: F_ref=4/5=0.80; F_cur=1/5=0.20. The gap is 0.60.
• At x=80: F_ref=5/5=1.00; F_cur=2/5=0.40. The gap is 0.60.
• Check all other observed jump points: no absolute gap exceeds 0.60. KS D = 0.60.

Step 2 — Compute 1D Wasserstein from sorted samples

• For two samples of equal length n with uniform weights, W1 = (1/n) sum_i |sorted_ref[i] − sorted_cur[i]|.
• Matched distances are |40−65|=25, |50−75|=25, |60−85|=25, |70−95|=25, and |80−105|=25, all in ₹ thousands.
• W1 = (25+25+25+25+25)/5 = 25 thousand rupees = ₹25,000.
• The result represents the average amount of distance needed to transport the empirical probability mass under optimal one-dimensional matching; it is not automatically an estimate of an individual customer's salary change.

Step 3 — Understand the KS p-value

• The two-sample KS null hypothesis is that independent observations come from the same continuous underlying distribution.
• With these tiny samples, SciPy's two-sided exact calculation gives p ≈ 0.357 for D=0.60, n_ref=n_cur=5.
• At a preselected 5% significance level, p > 0.05, so this tiny experiment does not reject the equal-distribution null. This does not establish that the distributions are equal.
• The ₹25,000 Wasserstein distance still conveys the practical magnitude of the observed shift; different questions need different evidence.

Reproduce with Python

PYTHON
import numpy as np
from scipy.stats import ks_2samp, wasserstein_distance

reference = np.array([40, 50, 60, 70, 80])
current = np.array([65, 75, 85, 95, 105])

ks = ks_2samp(reference, current, alternative="two-sided", method="exact")
w1 = wasserstein_distance(reference, current)

print("KS D:", ks.statistic)     # 0.6
print("KS p:", round(ks.pvalue, 3)) # 0.357
print("W1 in ₹ thousands:", w1) # 25.0

When should each method be used?

• Use KS when you need a dimensionless measure of the maximum cumulative separation and, under suitable sampling assumptions, a p-value for equality of continuous distributions.
• Use Wasserstein-1 when the feature's numeric scale matters—for example, how many ₹ thousands the income distribution moved—and you want a magnitude-sensitive distance.
• Use both when you need statistical evidence and an interpretable shift size. KS can see localized CDF discrepancies; Wasserstein accumulates differences over the numerical axis.
• Unlike PSI (Question 228), neither method requires manually defined bins. The main choice is instead how to validate significance, feature scales, reference periods and operational tolerances.

Important limitations

• Sample size matters: with very small samples, KS can have low power despite a substantial observed shift; on huge datasets, a tiny operationally unimportant shift can have a very small p-value.
• KS p-value assumptions: the familiar two-sample test assumes independent samples and a continuous null distribution. Many repeated values, discretized features or strong time dependence can make standard p-values unreliable; consider a suitable permutation/block-resampling design or other methods for those settings.
• Wasserstein has units and scale sensitivity: W1=25 is interpreted differently for income in ₹ thousands versus temperature in °C. If comparing features, use justified units or reference-based normalization and report what was done.
• Outliers and tails: Wasserstein may respond strongly to distant observations, while KS only uses the largest vertical ECDF difference and ignores the magnitude of distance along the x-axis.
• Neither is multivariate: a correlation change can go undetected when each feature's marginal distribution stays the same.
• Neither is concept-drift detection: shifts in P(Y|X) require outcome-based monitoring; a feature-distribution alert alone does not prove worse predictions.

Production monitoring workflow

• Fix data quality, timestamps and feature units first; select representative, sufficiently large, seasonally comparable reference and production windows.
• Compute KS D, its carefully interpreted p-value and W1 in the feature's units. Inspect ECDFs, quantiles, missingness, affected segments and possible outliers.
• Account for multiple testing across many monitored features; a flood of unadjusted p-values creates false positives.
• Calibrate action thresholds with historical variation, business impact and known safe changes; do not mechanically apply one W1 threshold to every feature.
• Investigate drift alerts alongside predictions and, when available, delayed ground-truth labels, model loss and business KPIs.
• Avoid automatic retraining solely because a KS test is significant or W1 is large. Diagnose the source and validate a candidate fix on recent representative outcomes.

Key takeaway: KS measures the maximum vertical difference between ECDFs; Wasserstein-1 measures the area between ECDFs and retains the feature's units. Report both the strength of statistical evidence and the practical size of a shift.

References

• SciPy: ks_2samp
• SciPy: wasserstein_distance