How do deep ensembles estimate uncertainty in classification?

Deep ensembles estimate predictive uncertainty by training several neural networks independently and combining their predicted class probabilities. The models may learn different plausible solutions, so their agreement—or disagreement—provides useful information beyond any one model's confidence.

How a deep ensemble is built

• Train multiple models: use the same classification task and training dataset, but train each network separately, typically with different random initializations and shuffled minibatches. Architecture may be shared; bootstrap samples or alternative architectures are optional diversity mechanisms.
• Retain separate models: do not average network weights; each trained network independently produces a class-probability vector.
• Combine at prediction time: for M members and class c, ensemble probability p̄(c|x) = (1/M) × Σₘ pₘ(c|x). Equal weights are a common baseline; learned weighting requires careful validation.
• Inspect uncertainty: use how concentrated the ensemble's *mean probabilities* are to describe predictive ambiguity, and how much member predictions differ to investigate model-related uncertainty.

Beginner-friendly numerical example: three image classifiers

Three separately trained models classify one image into Cat, Dog, Bird, returning:

• Model A: [0.80, 0.15, 0.05] — favors Cat.
• Model B: [0.60, 0.30, 0.10] — favors Cat.
• Model C: [0.40, 0.50, 0.10] — favors Dog.

Step 1 — Average the Cat probabilities: (0.80 + 0.60 + 0.40) / 3 = 0.6000.

Step 2 — Average Dog probabilities: (0.15 + 0.30 + 0.50) / 3 = 0.3167.

Step 3 — Average Bird probabilities: (0.05 + 0.10 + 0.10) / 3 = 0.0833.

Step 4 — Check the distribution: 0.6000 + 0.3167 + 0.0833 ≈ 1.0000. The ensemble predicts Cat, with a mean estimated probability of 60%.

Step 5 — Read the disagreement: Model C prefers Dog, unlike Models A and B. This is a practical sign that the classification may depend on which trained model is used. The averaged probability reflects this uncertainty better than blindly returning Model A's 80% Cat prediction—but averaging does not automatically guarantee a calibrated 60% risk estimate.

Contrast: If all three models instead output [0.80, 0.15, 0.05], the ensemble's mean is still [0.80, 0.15, 0.05], and there is no observed inter-model disagreement. That does not prove the image is easy or that every model is correct.

How does ensemble diversity help?

• Different random seeds, minibatch orders, and optional data resampling can lead models to different weight configurations and decision boundaries.
• When plausible models disagree, this may indicate epistemic uncertainty, related to limited training evidence or alternative model explanations. Diversity is a proxy, not a mathematically exact posterior uncertainty estimate.
• A set of nearly identical models provides little extra information. Conversely, arbitrary disagreement caused by poorly trained members is not automatically useful uncertainty.
• Compare probability vectors, not logits or predicted class IDs, when producing the ensemble mean. This preserves probability interpretation.

Python example

PYTHON
import numpy as np

# Each row is one independently trained model's class probabilities.
member_probs = np.array([
    [0.80, 0.15, 0.05],
    [0.60, 0.30, 0.10],
    [0.40, 0.50, 0.10],
])
labels = ["Cat", "Dog", "Bird"]
mean_probs = member_probs.mean(axis=0)
print(dict(zip(labels, np.round(mean_probs, 4))))
# {'Cat': 0.6, 'Dog': 0.3167, 'Bird': 0.0833}
print(labels[mean_probs.argmax()])  # Cat

Evaluation: is the uncertainty useful?

• Accuracy and proper probabilistic scores: evaluate on an untouched test set using accuracy, negative log-likelihood, and Brier score.
• Calibration: inspect reliability diagrams and an explicitly defined ECE variant; do not assume averaging creates calibrated probabilities.
• Error detection: test whether high disagreement or low ensemble confidence identifies actual mistakes, using AUROC or risk at matched coverage.
• Distribution shift and out-of-distribution examples: evaluate how uncertainty behaves on shifted, corrupted, or unfamiliar inputs. A model ensemble can still agree confidently on an unseen or systematically biased input.
• Compare baselines: evaluate a single model and an ensemble of various sizes on identical held-out data, including multiple training seeds if feasible.

Computational trade-offs and limitations

• Training cost: M independently trained members can require roughly M times the model training work, although separate training jobs are easy to parallelize.
• Inference cost: all M members may need forward passes, increasing compute, latency, and memory unless parallelized or compressed.
• Storage/deployment: M sets of model weights must be maintained, and updates must preserve compatibility across preprocessing and class labels.
• Diminishing returns: adding more similar members can offer little gain. Check validation results before increasing ensemble size.
• Correlated errors: all models may share the same dataset bias, architecture weakness, or distribution shift; disagreement alone cannot certify reliability.
• Applications: medical-image triage, quality inspection, image classification, and review queues where both prediction quality and awareness of uncertainty matter.

Key takeaway: Deep ensembles use independently trained models, average their class probabilities, and inspect member disagreement. This is a practical route to predictive uncertainty, but it is not a guarantee of Bayesian correctness or protection from distribution shift.

References

• Lakshminarayanan, Pritzel & Blundell (2017), Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Google Research, Can You Trust Your Model's Uncertainty?