What is temperature scaling for probability calibration?

Temperature scaling is a post-training method that adjusts a classifier's predicted probabilities using one positive parameter T, without changing the order of its class scores. It is useful when a model frequently makes predictions with more or less confidence than its correctness warrants.

How it works

• Logits are the raw scores output by a classification model before softmax.
• Convert logits z to probabilities with softmax(z / T): for class j, probability = exp(zⱼ/T) / Σₖ exp(zₖ/T).
• T = 1: original probabilities.
• T > 1: spreads probability more evenly across classes (softens confidence).
• 0 < T < 1: makes the largest logits more dominant (sharpens confidence).
• Because every logit is divided by the same positive number, the highest-scoring class is unchanged. Temperature scaling usually leaves hard-label accuracy unchanged, but changes probability-based metrics.

Beginner-friendly numerical example

Suppose a three-class image classifier produces logits [2, 1, 0] for [Cat, Dog, Bird].

Step 1 — Original probabilities (T = 1)

• Exponentials: exp(2) ≈ 7.389, exp(1) ≈ 2.718, exp(0) = 1; sum ≈ 11.107.
• Softmax ≈ [0.665, 0.245, 0.090]. The model predicts Cat with 66.5% confidence.

Step 2 — Soften with T = 2

• Adjusted logits: [1, 0.5, 0].
• Exponentials: approximately [2.718, 1.649, 1], sum ≈ 5.367.
• Softmax ≈ [0.506, 0.307, 0.186]. Cat remains the top prediction, but its confidence drops to 50.6%.

Step 3 — Sharpen with T = 0.5

• Adjusted logits: [4, 2, 0].
• Exponentials: approximately [54.598, 7.389, 1], sum ≈ 62.987.
• Softmax ≈ [0.867, 0.117, 0.016]. Cat confidence increases to 86.7%.

Key observation: Cat remains the top class at all three temperatures. The probabilities change, not the predicted class.

How is the temperature fitted?

1. Train the base classifier normally; freeze its model weights.
2. Prepare a separate, representative calibration/validation set not used to fit the classifier or evaluate its final test performance.
3. Compute the frozen model's logits for that set.
4. Find a single T > 0 that minimizes the mean negative log-likelihood (cross-entropy) of the true labels after softmax(z/T). Optimize log(T), or otherwise constrain T to remain positive.
5. Apply that same learned T to future logits at inference time; assess resulting probabilities on an untouched test set.

Simple PyTorch sketch:

PYTHON
import torch
import torch.nn.functional as F

# val_logits: [N, K] from a frozen model; val_labels: [N]
log_T = torch.nn.Parameter(torch.zeros(()))
optimizer = torch.optim.LBFGS([log_T], lr=0.1, max_iter=100)

def closure():
    optimizer.zero_grad()
    T = log_T.exp()
    loss = F.cross_entropy(val_logits / T, val_labels)
    loss.backward()
    return loss

optimizer.step(closure)
temperature = float(log_T.exp().detach())
# Inference: calibrated_prob = softmax(test_logits / temperature)

How does this relate to calibration metrics?

• Reliability diagrams help reveal whether confidence matches empirical accuracy in confidence regions.
• ECE, MCE, ACE, and TACE provide different summaries of the remaining calibration mismatch, depending on how scores are grouped or filtered.
• Brier score and negative log-likelihood evaluate full predictive probabilities. Minimizing validation log-loss does not guarantee improving every other calibration metric; verify on held-out data.
• Temperature scaling is a calibration method, not a new metric or a way to improve ranking.

Applications and limitations

• Applications: image classification, document labeling, and decision support systems where reliable class probabilities matter.
• One global parameter: a single T may not fix class-specific or subgroup-specific calibration errors.
• Ranking unchanged: temperature scaling cannot correct which class was ranked first or improve top-1 accuracy by itself.
• Data shift: a temperature learned on past validation data may not calibrate future inputs after distribution changes.
• Do not confuse with generation temperature: both change softmax sharpness, but fitting temperature to classification labels for probability calibration is a specific post-hoc procedure.

Key takeaway: Learn one temperature on held-out data to soften or sharpen class probabilities while preserving the predicted label.

References

• Guo et al. (2017), On Calibration of Modern Neural Networks
• scikit-learn Probability Calibration