What is probability calibration?

Probability calibration means checking whether a classifier's predicted probabilities match how often events actually occur. A model is well calibrated if, among examples assigned about 70% probability of the positive class, approximately 70% really are positive.

Why does calibration matter?

A classifier can rank cases correctly but still report misleading probabilities. For example, two models might identify the same patients as higher risk, while one consistently predicts 90% for cases that actually occur only 60% of the time. That model is overconfident.

• Overconfident: predicted probabilities are higher than observed event frequencies.
• Underconfident: predicted probabilities are lower than observed event frequencies.
• Well calibrated: predicted probabilities agree with observed frequencies, within sampling uncertainty.

Calibration is not the same as accuracy or ROC-AUC. Accuracy checks hard class labels; ROC-AUC assesses ranking. Calibration checks whether the numeric probabilities are trustworthy.

Beginner-friendly numerical example

A loan-default model predicts approximately 0.80 default probability for 20 borrowers. Suppose 12 of those borrowers actually default.

Average predicted default probability = 0.80
Observed default frequency = 12 / 20 = 0.60

Calibration gap in this group = 0.80 - 0.60 = 0.20

The model predicted an 80% event rate, but only 60% defaulted in this group—suggesting overconfidence by 20 percentage points. One small group cannot establish overall model calibration; assess multiple groups on enough held-out data.

How do you evaluate calibration?

1. Collect out-of-sample predicted probabilities and actual binary labels.
2. Group predictions into bins, such as around 0.1, 0.2, ..., 0.9.
3. For each bin, calculate the mean predicted probability and the observed positive fraction.
4. Plot these points in a reliability diagram (calibration curve). Ideal calibration follows the diagonal observed frequency = predicted probability.

Python example: reliability diagram

PYTHON
import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve

# Example probabilities and observed class labels
probs = [0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9]
y_true = [0, 0, 0, 1, 0, 1, 1, 1]

fraction_positive, mean_predicted = calibration_curve(
    y_true, probs, n_bins=4, strategy="uniform"
)

plt.plot(mean_predicted, fraction_positive, marker="o")
plt.plot([0, 1], [0, 1], "--", label="Perfect calibration")
plt.xlabel("Mean predicted probability")
plt.ylabel("Observed positive fraction")
plt.legend()
plt.show()

This tiny example demonstrates the *method*, not a statistically reliable assessment; use more observations for real evaluation.

Can you improve calibration?

Yes. A separate calibration stage can map model scores to probabilities using approaches such as sigmoid/Platt scaling, isotonic regression, or temperature scaling. Fit any calibration method using data separate from the examples used to train the base model, or a suitable cross-validation procedure, to avoid misleadingly optimistic results.

Important: metrics such as Brier score and log loss assess probability quality but also reflect discrimination and uncertainty; lower scores do not by themselves prove better calibration.

When does it matter most?

• Healthcare risk scores: people need to interpret predicted probabilities.
• Loan and fraud risk: thresholds may depend on estimated risk or expected costs.
• Forecasting and prioritization: downstream teams use probabilities to make decisions.

In short: a calibrated 80% prediction means about 80 out of 100 comparable cases should experience the event—not that any individual event is guaranteed.

Reference

• scikit-learn: Probability calibration