How can Jensen–Shannon divergence detect feature distribution drift?

Jensen–Shannon divergence (JSD) measures how different two probability distributions are. In ML monitoring, it compares the distribution of a categorical feature or consistently binned numerical feature between a reference window and current production. Unlike one-sided Kullback–Leibler (KL) divergence, JSD is symmetric and bounded.

Formula and interpretation

• Let P be the reference category probabilities and Q be the current probabilities, with nonnegative values summing to 1.
• Form their midpoint distribution M = (P + Q)/2, calculated independently for every category.
• KL(P || M) = sum_i P_i log₂(P_i/M_i) and KL(Q || M) = sum_i Q_i log₂(Q_i/M_i).
• JSD(P,Q) = 0.5 KL(P || M) + 0.5 KL(Q || M).
• Jensen–Shannon distance (JSDist) = sqrt(JSD). SciPy's scipy.spatial.distance.jensenshannon returns the distance, not the divergence.
• With log base 2, the divergence ranges from 0 to 1 bit, and the distance ranges from 0 to 1. Zero means identical distributions. One is the maximum when distributions have disjoint support.
• The formula is symmetric, so JSD(P,Q) = JSD(Q,P). Unlike PSI, JSD stays finite when a category appears in only one of the two distributions.

Worked numerical example: payment methods

A checkout model monitors Card, UPI, Cash across two nonoverlapping windows. Assume both samples are large enough for the proportions shown to be useful:

• Reference P = [0.60, 0.30, 0.10].
• Production Q = [0.20, 0.50, 0.30].

Step 1 — Midpoint distribution

• Card: (0.60 + 0.20)/2 = 0.40.
• UPI: (0.30 + 0.50)/2 = 0.40.
• Cash: (0.10 + 0.30)/2 = 0.20.
• Thus M = [0.40, 0.40, 0.20], summing to 1.

Step 2 — KL(P || M), in bits

• Card: 0.60 log₂(0.60/0.40) = +0.350978.
• UPI: 0.30 log₂(0.30/0.40) = −0.124511.
• Cash: 0.10 log₂(0.10/0.20) = −0.100000.
• Total KL(P || M) = 0.126466 bits.
• Individual KL contributions can be negative; the total KL remains nonnegative.

Step 3 — KL(Q || M), in bits

• Card: 0.20 log₂(0.20/0.40) = −0.200000.
• UPI: 0.50 log₂(0.50/0.40) = +0.160964.
• Cash: 0.30 log₂(0.30/0.20) = +0.175489.
• Total KL(Q || M) = 0.136453 bits.

Step 4 — JSD and Jensen–Shannon distance

• JSD = (0.126466 + 0.136453)/2 = 0.131460 bits.
• Jensen–Shannon distance = sqrt(0.131460) ≈ 0.362573.
• Interpretation: payment-method proportions changed; JSD summarizes their discrepancy. 0.13146 is not a 13.146% model-error rate, and it is not a p-value.

Python verification

PYTHON
import numpy as np
from scipy.special import rel_entr
from scipy.spatial.distance import jensenshannon

p = np.array([.60, .30, .10])
q = np.array([.20, .50, .30])
mid = (p + q) / 2

# rel_entr is in nats; divide by ln(2) to get bits.
kl_p = rel_entr(p, mid).sum() / np.log(2)
kl_q = rel_entr(q, mid).sum() / np.log(2)
js_div = (kl_p + kl_q) / 2
js_dist = jensenshannon(p, q, base=2)

print(np.round(mid, 3))  # [0.4, 0.4, 0.2]
print(round(kl_p, 6), round(kl_q, 6))  # 0.126466 0.136453
print(round(js_div, 6))   # 0.13146
print(round(js_dist, 6))  # 0.362573
assert np.isclose(js_dist**2, js_div)

Zeros, new categories, and feature types

• Zero proportions: define 0 log(0/a) = 0 by its limit. If P_i=0 and Q_i>0, then M_i>0, so the other KL contribution remains finite. If P_i=Q_i=0, their combined contribution is zero.
• You generally do not need pseudocount smoothing just to avoid infinite JSD. For very sparse categories, however, smoothing or category grouping may help estimate the underlying population distribution more stably; report and validate those choices.
• Categorical features: use stable shared labels, including Missing and New/Other buckets where appropriate; ensure the reference and production vectors align by category.
• Numerical features: create fixed histogram bins using a representative reference sample and retain identical edges for every monitoring window. Shifting bin boundaries invalidate direct comparison.
• Important limitation of histograms: changes inside a bin, or changes in relationships between features, may be invisible.

Thresholds and practical monitoring

• There is no universal JSD threshold such as “0.1 means retrain.” Some products use warning/alert cutoffs, but these are operational heuristics rather than distribution-free significance tests.
• Build baseline variation by comparing historical reference-like windows or resampling the reference data. Calibrate alerts to sample size, seasonality, category frequency, business impact, and the drift expected during normal operations.
• Compare the feature across comparable time windows and segments; inspect category percentages rather than relying exclusively on one score.
• When many features are tested, account for correlated alerts, multiple comparisons, noisy rare categories, and operational alert fatigue.
• A drift alert should trigger investigation into upstream data quality, category definitions, business behavior, and downstream model quality—not automatic retraining.

Applications, comparison, and limitations

• Applications: monitoring payment methods, user acquisition channels, traffic sources, country mix, model-score histogram shifts, or numerical features after reference-based binning.
• Versus PSI (Question 228): both use proportions, but PSI is a symmetric Jeffreys divergence that can become infinite for one-sided zeros; JSD is symmetric, finite and bounded with log₂.
• Versus KS/Wasserstein (Question 229): JSD is directly useful for categorical or binned distributions; KS and 1D Wasserstein use numerical ordering and avoid this particular histogram binning requirement.
• Versus chi-square (Question 230): chi-square tests a count-based null hypothesis using sampling assumptions and can report a p-value. JSD is a descriptive divergence, not a hypothesis-test p-value by itself.
• Sampling variability: a nonzero sample JSD can occur even if the true distributions are unchanged; small samples and rare categories produce unstable estimates.
• Not direct model-performance or concept-drift detection: a large JSD does not establish reduced accuracy, and a change in P(Y|X) can occur with unchanged feature marginals.
• Class-count and resolution dependence: a normalized 0–1 range does not guarantee that JSD scores from unrelated classification tasks, different category sets, or different histogram schemes are meaningfully interchangeable.

Key takeaway: JSD measures how much reference and current feature proportions differ, with a symmetric, finite, bounded score. Calculate divergence carefully, distinguish it from Jensen–Shannon distance, and use alerts to investigate rather than to mandate retraining.

References

• SciPy: Jensen–Shannon distance
• EY: AI Models Ongoing Monitoring discussion