What is Expected Calibration Error (ECE) in classification?

Expected Calibration Error (ECE) summarizes how closely a classifier's stated confidence matches its observed correctness. It splits predictions into confidence bins and averages the absolute confidence-versus-accuracy gaps, weighting each bin by its number of examples. Lower ECE is better, but zero estimated ECE does not guarantee perfect calibration.

Formula

ECE = Σ over bins [ (number of examples in bin / total examples) × |bin accuracy − mean bin confidence| ].

For standard top-label ECE, confidence is the highest predicted class probability, and accuracy means whether that highest-scoring class was correct.

Beginner-friendly numerical example

Suppose we have 10 predictions, grouped into three nonempty confidence bins:

• Bin A: two predictions, each with 55% confidence; 1/2 correct = 50% accuracy. Gap = |0.50 − 0.55| = 0.05; weight = 2/10 = 0.20; contribution = 0.010.
• Bin B: three predictions, each with 75% confidence; 2/3 correct ≈ 66.7% accuracy. Gap = |2/3 − 0.75| ≈ 0.0833; weight = 3/10 = 0.30; contribution = 0.025.
• Bin C: five predictions, each with 90% confidence; 4/5 correct = 80% accuracy. Gap = |0.80 − 0.90| = 0.10; weight = 5/10 = 0.50; contribution = 0.050.

Add all weighted gaps: ECE = 0.010 + 0.025 + 0.050 = 0.085, or 8.5 percentage points of average absolute bin-level calibration gap.

The value 8.5% is not classification error or accuracy. A small ECE can coexist with poor prediction accuracy, and opposite local deviations can be hidden by broad bins.

Python: calculate top-label ECE

PYTHON
import numpy as np

confidence = np.array([.55]*2 + [.75]*3 + [.90]*5)
correct = np.array([1,0, 1,1,0, 1,1,1,1,0])
edges = np.array([0.0, 0.70, 0.85, 1.0])

ece = 0.0
for left, right in zip(edges[:-1], edges[1:]):
    mask = (confidence >= left) & (
        (confidence < right) if right < 1.0 else (confidence <= right)
    )
    if mask.any():
        gap = abs(correct[mask].mean() - confidence[mask].mean())
        ece += mask.mean() * gap
print(round(ece, 3))  # 0.085

Binning choices and interpretation

• Equal-width bins have fixed probability intervals; equal-frequency (quantile) bins aim for similar sample counts.
• Too few bins can hide local miscalibration; too many bins create unstable observed accuracies.
• State the bin boundaries, empty-bin handling, sample size, and class distribution. Different choices can produce different ECE values even for the same model.

Top-label vs classwise ECE

• Top-label ECE considers only the predicted winning label's confidence and whether that prediction was correct.
• Classwise ECE assesses each class's predicted probability against whether the true label belongs to that class (one-vs-rest), then aggregates across classes.
• Top-label calibration does not establish calibration of every class probability; the two definitions can disagree, and their numerical values should be identified explicitly.

Limitations and applications

• Finite-sample bias and instability: ECE is a binned estimate, not an exact population calibration error. Report the number of samples and consider sensitivity analyses or uncertainty intervals.
• Cancellation inside bins: over- and underconfidence can cancel before taking the absolute gap, making ECE deceptively low.
• Rare classes: aggregate top-label ECE can mask failure on minority classes; inspect classwise and subgroup calibration.
• Not a discrimination metric: ECE does not measure whether positive examples are ranked ahead of negatives. Pair it with suitable performance metrics.
• Applications: monitoring confidence quality for image classifiers, risk classifiers, and human-review workflows where confidence affects decisions.

Key takeaway: ECE is the sample-weighted average of absolute gaps between mean confidence and observed accuracy in confidence bins. Always report which ECE variant and binning procedure you used.

References

• Survey of uncertainty in deep neural networks
• Expected Calibration Error definition and classwise variants