What is variation ratio in classification uncertainty estimation?

Variation ratio (VR) is a simple measure of disagreement among categorical predictions collected for the *same input* from repeated stochastic model passes or multiple models. It asks what fraction of the predicted labels do not match the most frequently predicted label (the mode).

Formula and interpretation

• Let T be the number of predicted-label samples for one input.
• Find the modal class: the label receiving the greatest number of votes.
• Let f_mode be how many votes that class receives.
• VR = 1 − f_mode/T.
• VR = 0: all predictions vote for the same class.
• Larger VR: votes are more dispersed among labels. For C classes, the theoretical maximum is 1 − 1/C, achieved by equal class frequencies when feasible. For two classes, the maximum is 0.5.
• For equal-count ties, the modal class may not be unique, but f_mode (the maximum vote count) still is.

Step-by-step example: eight stochastic predictions

A classifier examines the same animal photo eight times using an ensemble or Monte Carlo dropout. Its predicted labels are:

• Passes 1–8: Cat, Dog, Cat, Bird, Cat, Dog, Cat, Dog.
• Step 1 — Count votes: Cat = 4, Dog = 3, Bird = 1.
• Step 2 — Find the mode: Cat received the most votes; f_mode = 4.
• Step 3 — Calculate: VR = 1 − 4/8 = 0.50.
• Step 4 — Interpret: 50% of votes disagree with Cat, indicating substantial variation among the eight categorical predictions. This is not a claim that the probability of being wrong is 50%.

A useful contrast

• If all eight passes vote Cat, VR = 1 − 8/8 = 0, regardless of how close their underlying softmax probabilities are to a tie.
• If three classes receive votes 3, 3, 2, then VR = 1 − 3/8 = 0.625. The most common class need not have a majority.
• Variation ratio is based on the chosen categorical label samples, not on the full predicted probability vectors.

Python example

PYTHON
from collections import Counter

votes = ["Cat", "Dog", "Cat", "Bird", "Cat", "Dog", "Cat", "Dog"]
counts = Counter(votes)
modal_count = max(counts.values())
variation_ratio = 1 - modal_count / len(votes)
print(counts)                 # Cat: 4, Dog: 3, Bird: 1
print(variation_ratio)        # 0.5

How do you obtain the votes?

• Deep ensembles: collect a predicted class from each independently trained model.
• Monte Carlo dropout: run multiple stochastic forward passes on the same input with dropout enabled but no training updates; record each resulting categorical prediction.
• Important implementation choice: traditional Bayesian VR definitions sample a class label from each predicted softmax distribution. Other implementations take the argmax label of each pass. These two procedures can give different VR values, so explicitly state which was used.

Variation ratio versus mutual information (Question 221)

• VR counts class votes: it ignores how confident a model was when casting each vote. It is easy to compute but coarse.
• Mutual information compares probability distributions: it subtracts mean per-model entropy from entropy of their average to quantify model-related disagreement.
• For example, models that all predict Cat with probabilities [0.51, 0.49] yield argmax-based VR = 0, even though their probabilities are nearly tied. MI also depends on *variation across models*, not merely individual ambiguity, so the metrics capture different properties.
• VR is a vote-dispersion indicator, not a complete measure or guaranteed isolation of epistemic uncertainty.

Applications and limitations

• Active learning: identify cases with unstable class labels for annotation, then test whether this improves learning.
• Human review: prioritize disputed predictions in document classification, image inspection, or quality control.
• Error/OOD detection: validate VR as a score against actual mistakes and unfamiliar inputs; consider AUROC and selective risk at matched coverage.
• Sample-count sensitivity: with only a few passes, votes and VR are noisy. Study robustness to ensemble size or MC samples.
• Class-count dependence: VR ranges differ with the number of classes; report C, T, and the vote-generation rule.
• Shared blind spots: models can agree confidently on the wrong class, giving VR = 0.
• Lost probability detail: VR cannot distinguish a narrow win from an overwhelming win when both produce the same modal count.

Key takeaway: Variation ratio measures how often stochastic predicted labels disagree with the modal label, rather than how spread out their full probability distributions are.

References

• Yarin Gal, Uncertainty in Deep Learning, section on variation ratios
• Gal, Islam & Ghahramani (2017), Deep Bayesian Active Learning with Image Data