What is the difference between calibration and discrimination in classification?

Calibration asks whether predicted probabilities match event frequencies. Discrimination asks whether the model ranks examples that experience the event ahead of those that do not. A model can be strong on one property and weak on the other.

Beginner-friendly numerical example: two classifiers, six customers

Actual outcomes (churn = 1, no churn = 0) for Customers 1–6: [1, 1, 1, 0, 0, 0].

Model A — calibrated overall, but weak discrimination

• Gives every customer 0.50 predicted churn probability: [.50, .50, .50, .50, .50, .50].
• Calibration: among the six 50%-confidence predictions, 3/6 = 50% actually churn. Thus it is perfectly calibrated *on this single observed group*.
• Discrimination: all customers are tied. ROC-AUC = 0.50 using the usual half-credit convention for tied positive–negative pairs.
• Model A offers no useful ordering for prioritizing customers despite matching the group event rate.

Model B — perfect discrimination, poorly calibrated probabilities

• Gives churners 0.60 and non-churners 0.40: [.60, .60, .60, .40, .40, .40].
• Discrimination: every positive score exceeds every negative score. There are 3 × 3 = 9 positive–negative pairs, all correctly ordered, so ROC-AUC = 9/9 = 1.00.
• Calibration: the 60%-score group actually churns 3/3 = 100%; the 40%-score group churns 0/3 = 0%. Both groups' event frequencies differ from their predicted probabilities.
• Model B ranks customers perfectly here but underestimates risk for churners and overestimates it for non-churners in these two groups.

Important: With just six observations, these are illustrative *sample-level* findings, not reliable evidence of population calibration.

Which metrics evaluate each property?

• Calibration: reliability diagrams, ECE/MCE and their variants describe probability–frequency gaps; Brier score and log loss assess overall probabilistic quality, but also reflect discrimination/resolution.
• Discrimination: ROC-AUC measures pairwise ranking across positive and negative cases; PR-AUC/average precision can be especially informative when positives are rare.
• Decision usefulness: threshold-specific precision, recall, false-positive rates, workload, and error costs answer operational questions neither property resolves alone.

Practical engineering implications

• Risk ranking: for deciding which customers to contact first, ranking ability matters.
• Risk estimates: for reporting “this customer has a 70% churn risk,” credible probabilities matter.
• Both matter: a financial or medical triage workflow may need both reliable risk ordering and probabilities for resource allocation.
• Post-hoc calibration: strictly increasing mappings such as positive-slope sigmoid calibration can change probabilities without changing ROC-AUC; flat isotonic mappings can create ties and alter ranking metrics.
• Data shift and class prevalence: calibrate and evaluate on data representative of the intended deployment; both properties may change over time.

Python check

PYTHON
from sklearn.metrics import roc_auc_score

actual = [1, 1, 1, 0, 0, 0]
model_a = [.50] * 6
model_b = [.60, .60, .60, .40, .40, .40]

print(roc_auc_score(actual, model_a))  # 0.5
print(roc_auc_score(actual, model_b))  # 1.0
# Inspect observed rates in each distinct probability group
# to evaluate this example's calibration.

Limitations

• One metric is not enough: excellent ROC-AUC does not guarantee trustworthy probabilities, and low ECE need not imply useful ranking.
• Small calibration groups: perfect-looking agreement on few examples can be chance; use adequate support and confidence intervals when possible.
• Calibration depends on grouping: a population-average 50% forecast may be calibrated at that one score while unhelpful for individual prioritization.
• Operational costs differ: neither calibration nor ROC-AUC alone tells you the optimal action threshold.

Key takeaway: Calibration is about the numerical meaning of probabilities; discrimination is about the ordering of cases. Inspect both when predicted risks drive decisions.

References

• scikit-learn: Probability calibration
• scikit-learn: ROC-AUC score
• scikit-learn: model evaluation