What is a calibrated classifier?

A calibrated classifier is a classification model whose output probabilities are adjusted or learned so that they better match observed event frequencies. When it predicts about 70% risk for a sufficiently large group of similar cases, roughly 70% of those cases should experience the event.

What makes it different from an ordinary classifier?

• A classifier may correctly rank examples from low risk to high risk while still giving misleading probability values.
• A calibrated classifier produces probabilities intended to be meaningful for risk-based decisions, not just scores for ranking.
• Calibration is evaluated on held-out data: training-set confidence alone is not proof of calibration.

Beginner-friendly numerical example

Suppose a credit-risk classifier assigns a default probability of 0.80 to 50 borrowers in a validation group. Only 30 actually default:

Original predicted default probability = 80%
Observed default frequency = 30 / 50 = 60%

This suggests the model is overconfident in this group. A suitable calibration mapping may adjust such predictions from 0.80 toward 0.60, assuming the pattern is supported by enough representative calibration data.

The mapping is learned from many examples, not by manually subtracting 20 percentage points from every prediction. Calibrated outputs also are not guaranteed to equal observed outcomes in every small subgroup.

How do you build one?

1. Fit a base classifier to training data. It might generate decision scores or uncalibrated probabilities.
2. Obtain out-of-sample predictions from the base classifier for calibration.
3. Fit a calibration mapping that converts the model's outputs into better probability estimates.
4. Evaluate the calibrated model on a separate test set, checking reliability diagrams, log loss or Brier score, and the task's decision metrics.

Common calibration mappings include:

• Sigmoid (Platt scaling): fits a smooth sigmoid mapping; a useful default for limited calibration data.
• Isotonic regression: learns a flexible nondecreasing mapping but can overfit with small calibration datasets.
• Temperature scaling: adjusts logits before converting them to probabilities, often for neural-network classification.

Python example: calibrate a linear SVM

PYTHON
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import brier_score_loss

X, y = make_classification(
    n_samples=1000, n_features=6, n_informative=4,
    n_redundant=0, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

base = make_pipeline(StandardScaler(), LinearSVC())
calibrated = CalibratedClassifierCV(
    estimator=base, method="sigmoid", cv=5
)
calibrated.fit(X_train, y_train)

prob_positive = calibrated.predict_proba(X_test)[:, 1]
print("Brier score:", round(brier_score_loss(y_test, prob_positive), 3))

Cross-validation ensures that calibration uses predictions from folds not used to fit that fold's base model. Scaling is inside the pipeline so preprocessing is also fit only on the relevant training fold.

What should you watch out for?

• Calibration is not classification accuracy: a well-ranked classifier can be poorly calibrated and vice versa.
• Calibrating does not always improve test performance. Validate the change on unseen data and consider sampling uncertainty.
• Calibration and thresholds are different: calibrated probabilities can support cost-based decisions, but you still choose a threshold separately.
• Changing ranking is possible: strictly monotonic sigmoid mappings preserve within-model rank order, while isotonic mappings can create tied scores.

In short: a calibrated classifier provides probabilities that can be interpreted as estimated event frequencies, enabling more reliable decisions when actual risk matters.

References

• scikit-learn: Probability calibration
• scikit-learn: CalibratedClassifierCV