What is mutual information for estimating epistemic uncertainty in classification?

Mutual information (MI) in Bayesian predictive classification estimates how much a predicted label tells us about uncertain model parameters. A common estimator measures the disagreement among predictions from a Bayesian model's parameter samples, a deep ensemble, or Monte Carlo dropout. The acquisition score is often called BALD (*Bayesian Active Learning by Disagreement*).

Formula

For M sampled predictive probability vectors p⁽¹⁾, …, p⁽ᴹ⁾ for one example:

• Mean predictive distribution: p̄ = (1/M) Σₘ p⁽ᵐ⁾.
• Predictive entropy: H(p̄), measuring total uncertainty in the mixture.
• Mean individual entropy: (1/M) Σₘ H(p⁽ᵐ⁾), measuring average uncertainty *within* sampled models.
• Mutual information: MI ≈ H(p̄) − (1/M) Σₘ H(p⁽ᵐ⁾).
• Here H(p) = −Σ꜀ p꜀ log₂ p꜀, in bits. Always use a consistent log base.

Worked example: two models, two classes

Suppose two plausible classifiers evaluate the same picture as Cat versus Dog:

• Model A: [Cat 0.90, Dog 0.10].
• Model B: [Cat 0.10, Dog 0.90].

Step 1 — Average their probabilities:

• p̄ = [(0.90+0.10)/2, (0.10+0.90)/2] = [0.50, 0.50].

Step 2 — Entropy of the averaged distribution:

• H(p̄) = −0.5 log₂ 0.5 − 0.5 log₂ 0.5 = 1.000 bit.

Step 3 — Entropy within each model:

• H(Model A) = −0.9 log₂ 0.9 − 0.1 log₂ 0.1 ≈ 0.469 bits.
• Model B has the same entropy, 0.469 bits.
• Mean individual entropy = (0.469+0.469)/2 = 0.469 bits.

Step 4 — Subtract:

• MI = 1.000 − 0.469 ≈ 0.531 bits.
• Interpretation: each model is relatively confident, but they disagree about the class. That disagreement contributes substantial model-related uncertainty.

Contrast: uncertain models that agree

Suppose *both* models predict [0.50, 0.50]:

• Entropy of their average = 1 bit.
• Mean individual entropy = 1 bit.
• MI = 0 bits even though predictive entropy remains high.
• This distinguishes *uncertainty within every model* from *uncertainty arising because models disagree*. It is an illustrative contrast, not an automatic identification of the exact source of real-world data noise.

How is MI estimated in practice?

• Deep ensemble: train several models with different initialization or training randomness; collect every model's class probability vector for the same input.
• Monte Carlo dropout: with an appropriately trained dropout network, enable dropout for stochastic inference passes (without updating weights), collect many probability vectors, and use the formula above.
• Bayesian parameter samples: sample model parameters from an approximate or exact posterior where feasible.
• Average probabilities, not class IDs or raw logits. Calculate entropy of the mean, then subtract mean entropy—those operations cannot be swapped.

Python example

PYTHON
import numpy as np
from scipy.stats import entropy

predictions = np.array([[.90, .10], [.10, .90]])
mean_prob = predictions.mean(axis=0)
total = entropy(mean_prob, base=2)
within = entropy(predictions, axis=1, base=2).mean()
mi = total - within

print(round(total, 3))   # 1.000
print(round(within, 3))  # 0.469
print(round(mi, 3))      # 0.531

Applications and limitations

• Active learning: prioritize examples where plausible models disagree, because obtaining their labels may be informative for learning.
• Review and monitoring: flag disputed predictions for human inspection or additional training data, then validate whether MI actually predicts errors.
• Approximation quality: ensembles and dropout samples are *proxies* for posterior uncertainty, not necessarily calibrated Bayesian posterior samples. MI may miss uncertainty when models share biases.
• Finite samples: few stochastic predictions lead to noisy estimates. Test robustness to model count and random seeds.
• Entropy is not automatically aleatoric uncertainty: the mean per-model entropy is sometimes used as an approximation to data uncertainty, but it may also reflect model misspecification or approximation issues.
• Comparability and safety: MI depends on class count and probability quality. Report class count and log base, evaluate on shifted data and relevant subgroups, and do not treat high MI as a guarantee that a new label will improve a model.

Key takeaway: Predictive entropy measures uncertainty of the averaged prediction; mutual information subtracts the uncertainty within individual model predictions to highlight model disagreement.

References

• Gal, Islam & Ghahramani (2017), Deep Bayesian Active Learning with Image Data
• Gal & Ghahramani (2016), Dropout as a Bayesian Approximation