What is Maximum Calibration Error (MCE) in classification?

Maximum Calibration Error (MCE) measures the largest absolute difference between average predicted confidence and observed accuracy among nonempty confidence bins. It helps locate the worst *observed bin-level* mismatch, rather than the average mismatch.

Formula

MCE = max over nonempty bins |bin accuracy − mean bin confidence|.

For standard top-label MCE, confidence is the probability assigned to the predicted class, and accuracy is the fraction of predictions that are correct in that bin. A smaller MCE is better; 0 means all examined bins have matching observed accuracy and average confidence, not necessarily that the model is perfectly calibrated everywhere.

Beginner-friendly numerical example

Consider 10 held-out predictions divided into three confidence bins:

• Bin A (5 predictions): mean confidence 60%, 3 of 5 correct → accuracy 60%; absolute gap = |0.60 − 0.60| = 0.00.
• Bin B (3 predictions): mean confidence 75%, 2 of 3 correct → accuracy 66.7%; absolute gap = |0.75 − 2/3| ≈ 0.0833.
• Bin C (2 predictions): mean confidence 90%, 1 of 2 correct → accuracy 50%; absolute gap = |0.90 − 0.50| = 0.40.

Step 1: Calculate each bin's absolute gap: 0.00, 0.0833, 0.40.

Step 2: Select the largest: MCE = max(0, 0.0833, 0.40) = 0.40, or 40 percentage points.

Step 3: Locate the problem: Bin C appears severely overconfident—its mean confidence is 90%, but only half of its two predictions are correct. Because it contains just two observations, do not treat this estimate as a reliable population-level worst-case guarantee.

MCE versus ECE

• MCE: only the largest bin gap matters, irrespective of that bin's size.
• ECE: the sample-count-weighted average of absolute bin gaps matters.
• For these same ten predictions, ECE = (5/10 × 0) + (3/10 × 0.0833) + (2/10 × 0.40) = 0.105, or 10.5 percentage points, whereas MCE = 40 percentage points.
• Both are estimates based on binning; they answer different questions: “worst observed confidence region?” versus “average observed discrepancy?”

Python example

PYTHON
import numpy as np

confidence = np.array([.60]*5 + [.75]*3 + [.90]*2)
correct = np.array([1,1,1,0,0, 1,1,0, 1,0])
edges = np.array([0.0, 0.70, 0.85, 1.0])

gaps, ece = [], 0.0
for left, right in zip(edges[:-1], edges[1:]):
    in_bin = (confidence >= left) & (
        (confidence < right) if right < 1 else (confidence <= right)
    )
    if not in_bin.any():
        continue
    gap = abs(correct[in_bin].mean() - confidence[in_bin].mean())
    gaps.append(gap)
    ece += in_bin.mean() * gap

print(round(max(gaps), 3))  # MCE: 0.4
print(round(ece, 3))        # ECE: 0.105

Binning sensitivity and practical use

• Few bins: large bins can average away overconfidence and underconfidence, hiding local problems.
• Many bins: small counts make accuracy estimates volatile; one incorrect prediction may produce an outsized MCE.
• Uniform versus quantile bins: changing the edges or using approximately equal-count bins may change which region has the largest gap.
• Inspect sample counts: report the worst bin's support and actual gap, not just the final MCE.
• Applications: risk assessment, human-review systems, and selective classification where a badly calibrated confidence range can be consequential.

Limitations

• Not a true worst-case bound: a finite-sample binned MCE is only the maximum gap *among the observed bins*, not proof that every possible confidence value is within that gap.
• No direction in the score: absolute values hide whether the model overestimates or underestimates; inspect each signed gap.
• Not an accuracy or ranking measure: also evaluate discrimination, error costs, and performance on important subgroups.
• Variant matters: top-label, binary-positive-class, and classwise calibration may have different MCE definitions; state which one is used.

Key takeaway: MCE finds the largest observed bin-level calibration discrepancy, but its reliability depends heavily on how many examples each bin contains.

References

• Guo et al., On Calibration of Modern Neural Networks
• Classifier calibration: a survey on how to assess and improve predicted class probabilities