What is Adaptive Calibration Error (ACE) in classification?

Adaptive Calibration Error (ACE) measures how far predicted probabilities differ from observed outcome frequencies, using equal-frequency (approximately equal-count) bins instead of fixed-width probability intervals. The original ACE proposed by Nixon et al. (2019) evaluates each class's predicted probabilities, not only the winning class.

Beginner-friendly numerical example: adaptive bins

Imagine six binary event probabilities, with true outcomes 0 (no event) or 1 (event):

• Example 1: probability 0.10, outcome 0.
• Example 2: probability 0.20, outcome 0.
• Example 3: probability 0.30, outcome 1.
• Example 4: probability 0.70, outcome 1.
• Example 5: probability 0.80, outcome 1.
• Example 6: probability 0.90, outcome 1.

Step 1 — Sort scores. They are already in ascending order. Choose two equal-frequency bins, each containing three examples.

Step 2 — First bin (0.10, 0.20, 0.30):

• Mean predicted probability = (0.10 + 0.20 + 0.30)/3 = 0.20.
• Observed positive rate = 1/3 ≈ 0.333.
• Absolute calibration gap = |0.333 − 0.20| ≈ 0.133.

Step 3 — Second bin (0.70, 0.80, 0.90):

• Mean probability = (0.70 + 0.80 + 0.90)/3 = 0.80.
• Observed positive rate = 3/3 = 1.00.
• Absolute gap = |1.00 − 0.80| = 0.20.

Step 4 — Average the two gaps. Adaptive-bin error = (0.1333 + 0.20)/2 ≈ 0.1667, or 16.7 percentage points. This is a *binary positive-class adaptive-binning illustration*, not a claim that all definitions of multiclass ACE return this value.

ACE formula and variants

For K classes and R adaptive bins per class, the original classwise ACE is:

ACE = (1 / (K × R)) × Σ over classes k and bins r |observed class-k frequency − mean predicted class-k probability|.

• Binary positive-class adaptive ECE: apply equal-frequency binning only to the positive-class probabilities and average bin gaps; with equal counts it gives the worked example above.
• Top-label adaptive ECE / ACE variant: sort each example's *largest* predicted class probability, and compare bin confidence against top-label correctness.
• Original multiclass ACE: for each class, separately sort *all* probabilities assigned to that class; bin and compare them with class-indicator outcomes, then average across classes and bins.
• These variants answer slightly different questions and should not be treated as interchangeable. Explicitly name the variant in reported results.

Fixed-width versus adaptive bins

• Fixed-width binning: bins have predefined ranges (for example 0–0.2, 0.2–0.4, and so on), which can be empty or sparsely populated when scores cluster.
• Adaptive (quantile/equal-frequency) binning: bin boundaries follow the score distribution so groups contain similar numbers of observations.
• Adaptive bins can make estimates more stable in high-density regions, but their probability widths differ and they can hide calibration variation in broad sparse regions.

Python example: binary positive-class adaptive-binned error

PYTHON
import numpy as np

probs = np.array([.10, .20, .30, .70, .80, .90])
actual = np.array([0, 0, 1, 1, 1, 1])
indices = np.argsort(probs, kind="stable")
groups = np.array_split(indices, 2)

gaps = [abs(actual[g].mean() - probs[g].mean()) for g in groups]
print(round(float(np.mean(gaps)), 4))  # 0.1667

Applications and limitations

• Useful for auditing probability outputs for image recognition, risk prediction, and other confidence-sensitive classifiers.
• Number of bins matters: results change with bin count, ties, small datasets, and implementation choices.
• Rare classes and tiny probabilities: original multiclass ACE may be dominated by many near-zero class probabilities; thresholded adaptive calibration error (TACE) was proposed to address this issue.
• Averages hide localized failures: inspect reliability plots and class/subgroup behavior, not just the final number.
• No accuracy guarantee: a low ACE does not imply good discrimination or useful decisions.

Key takeaway: ACE adapts bins to the *distribution of predicted probabilities*. When using the term ACE, specify whether it is original classwise ACE, a top-label adaptive variant, or binary positive-class adaptive calibration error.

References

• Nixon et al. (2019), Measuring Calibration in Deep Learning
• TorchUncertainty AdaptiveCalibrationError documentation