What is Thresholded Adaptive Calibration Error (TACE) in classification?

Thresholded Adaptive Calibration Error (TACE) measures class-probability calibration after discarding very small predicted probabilities, then grouping the remaining scores into approximately equal-count bins. Nixon et al. proposed thresholding because modern many-class models often assign near-zero probabilities to most classes, which can dominate ordinary ACE.

What happens in TACE?

• For each class, collect its predicted probabilities across examples and the corresponding one-versus-rest outcomes: 1 if that class is the true label, otherwise 0.
• Filter: retain only probabilities above a chosen threshold ε (for example, ε = 0.05). This is a *measurement filter*, not the threshold used to predict the winning class.
• Adaptively bin: sort the retained scores and split them into approximately equal-sized groups.
• Measure: calculate the absolute difference between each bin's mean probability and its observed class frequency.
• Aggregate: average the gaps across valid bins and classes. Specify how you handle too few retained values, empty bins, and classes with no surviving predictions; implementations can differ.

Beginner-friendly numerical example: one class

A three-class model produces the following six probabilities for class A, with class-A ground truth indicators:

• Example 1: probability 0.01, true A? 0.
• Example 2: probability 0.03, true A? 0.
• Example 3: probability 0.10, true A? 0.
• Example 4: probability 0.40, true A? 0.
• Example 5: probability 0.60, true A? 1.
• Example 6: probability 0.80, true A? 1.

Step 1 — Filter with ε = 0.05. Discard 0.01 and 0.03; keep 0.10, 0.40, 0.60, 0.80.

Step 2 — Create two adaptive bins, each with two surviving predictions:

• Lower bin: [0.10, 0.40]; mean confidence = 0.25; actual class-A frequency = 0/2 = 0; gap = |0.25 − 0| = 0.25.
• Higher bin: [0.60, 0.80]; mean confidence = 0.70; class-A frequency = 2/2 = 1.00; gap = |0.70 − 1.00| = 0.30.

Step 3 — Average this class's bin gaps: (0.25 + 0.30)/2 = 0.275, or 27.5 percentage points.

Important: 0.275 is the class-A contribution/average in this toy example, *not the model's full multiclass TACE*. Repeat the process for classes B and C, then average valid classwise results to obtain a complete multiclass estimate.

How does TACE differ from ACE?

• ACE: adaptively bins *all* class probabilities, including many near-zero values.
• TACE: filters out probabilities at or below ε *before* adaptive binning, so the evaluation focuses on class probabilities large enough to matter for the chosen threshold.
• TACE is not necessarily smaller or better than ACE: it measures calibration over a different subset of scores, so the values are not directly comparable without stating their definitions and thresholds.

Simple Python demonstration for class A

PYTHON
import numpy as np

p_a = np.array([.01, .03, .10, .40, .60, .80])
is_a = np.array([0, 0, 0, 0, 1, 1])
epsilon = .05
keep = np.flatnonzero(p_a > epsilon)
ordered = keep[np.argsort(p_a[keep], kind="stable")]
bins = np.array_split(ordered, 2)
gaps = [abs(p_a[idx].mean() - is_a[idx].mean()) for idx in bins]
print(round(float(np.mean(gaps)), 3))  # 0.275
# For full multiclass TACE, repeat for each class and aggregate.

Advantages, limitations, and applications

• Useful for many-class networks: avoids letting enormous numbers of tiny probabilities overwhelm the adaptive-bin calculation.
• Threshold sensitivity: changing ε changes which predictions contribute and can change the metric substantially. Report ε, bin count, sample size, and class coverage.
• Can omit rare-class evidence: higher thresholds can remove almost every probability for a rare class; do not silently label missing class-bin results as perfect calibration.
• Binning remains noisy: sparse surviving scores, ties, and wide adaptive intervals can make estimates unstable or hide local problems.
• Does not assess all confidence values: a low TACE says little about excluded probabilities, classification accuracy, or minority-group outcomes.
• Applications: auditing classwise confidence for image classifiers and other multi-category models, especially when thousands of non-winning class probabilities are tiny.

Key takeaway: TACE = classwise adaptive calibration assessment after filtering near-zero probabilities. It improves focus, but the score is meaningful only alongside the threshold and binning procedure.

References

• Nixon et al. (2019), Measuring Calibration in Deep Learning
• Extended paper (arXiv:1904.01685)