Machine Learning
Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.
What is a multilabel confusion matrix?
A multilabel confusion matrix gives a separate 2×2 confusion matrix for each possible label. Each label is treated as present (positive) or absent (negative), because one sample can contain multiple labels at once. Worked example: 4 photos, 2 possible tags For each photo, compare the actual and predicted tag sets: Photo 1: actual {Dog, Outdoor}; predicted {Dog, Outdoor}. Photo 2: actual {Dog}; predicted {Outdoor}. Photo 3: actual {Outdoor}; predicted {Outdoor}. Photo 4: actual {}; predicted {Dog}. Step 1 — Evaluate the Dog label independently TP = 1: Photo 1 actually has Dog and predicts Dog. FP = 1: Photo 4 predicts Dog but does not contain Dog. FN = 1: Photo 2 contains Dog but the model misses it. TN = 1: Photo 3 neither contains nor predicts Dog. The four counts total 4 photos . Step 2 — Evaluate the Outdoor label independently TP = 2: Photos 1 and 3 contain Outdoor and predict it. FP = 1: Photo 2 predicts Outdoor, although it is absent. FN = 0: no actual Outdoor tag is missed. TN = 1: Photo 4 neither contains nor predicts Outdoor. Again, the four counts total 4 photos . How is this different from a multiclass confusion matrix? A single label multiclass task normally has one K×K matrix showing confusion between mutually exclusive classes. A multilabel task has K separate 2×2 matrices , one per label. Each matrix counts that label's present/absent decisions independently. Consequently, one photo can contribute to several label specific matrices. Do not add all their counts and interpret the sum as the number of distinct photos. Python example Applications and limitations Useful for diagnosis: discover which tags are frequently missed or wrongly added, and calculate label specific precision and recall when needed. Watch for rare labels: a label with very few positives may have many TNs but still perform poorly on actual positives. Report support alongside the counts. Thresholds matter: changing a label's decision threshold changes its TP/FP/FN/TN counts. Does not capture label relationships: separate matrices cannot tell you whether two labels are consistently predicted together incorrectly, or whether a whole label set is exactly right. Zero support: if a label never occurs in the evaluation set, interpret its positive class metrics with care. Key takeaway: In multilabel evaluation, ask “For this label , which photos are true positives, false positives, false negatives, and true negatives?” and repeat for every label. Reference: scikit learn multilabel confusion matrix.
What are top-k accuracy and top-k error in classification?
Top k accuracy measures the proportion of examples whose correct class appears among the model's k highest scoring classes . Unlike top 1 accuracy, the highest ranked class does not have to be correct if another of the top k suggestions is. Step by step numerical example Imagine an image classifier with six possible labels: A, B, C, D, E, and F . Across six images, the true label appears at these ranks in descending model score order: Image 1: true label at rank 1 . Image 2: true label at rank 2 . Image 3: true label at rank 3 . Image 4: true label at rank 4 . Image 5: true label at rank 5 . Image 6: true label at rank 6 . Step 1 — Top 1: only Image 1 succeeds → 1/6 = 16.7% accuracy , 83.3% top 1 error . Step 2 — Top 3: Images 1, 2, and 3 succeed → 3/6 = 50% accuracy , 50% top 3 error . Step 3 — Top 5: Images 1–5 succeed → 5/6 = 83.3% accuracy , 16.7% top 5 error . Formula and interpretation Top k accuracy = number of samples with their true class among the k highest scores / total samples. Top k error = 1 − top k accuracy (when both are fractions). If scores are fixed, top 1 ≤ top 3 ≤ top 5 accuracy for valid k values. Raising k can make the reported accuracy look better without actually improving the classifier. When k equals the total number of classes , top k accuracy is trivially 100%; it is not informative. Python example When is top k useful? Image classification: allow a few guesses among many visually similar categories. Assisted categorization or diagnosis: measure whether the right option is included among several suggestions a human will review. Large class models: compare ranking quality when a single top prediction is too restrictive. Limitations Single label setting: the standard scikit learn top k metric expects one true class per example; multilabel classification requires separate evaluation choices. K must be meaningful: top 5 is uninformative for a five class problem and top k scores from different k values or class counts should not be compared directly. No confidence assessment: being in the top k does not imply a well calibrated probability or a safe recommendation. Ties and ordering: tied class scores can change rankings depending on implementation; scikit learn uses higher class indices first for ties. Keep score columns aligned with the explicit labels order. Unequal class frequency: report per class results where rare or high impact labels matter. Key takeaway: Top k accuracy asks “Did the true class appear in the model's first k suggestions?” Top k error is the complementary miss rate. References: scikit learn top k accuracy score and Google ML glossary.
What is Cohen's kappa score in classification?
Cohen's kappa (κ) measures agreement between two sets of class labels after accounting for agreement expected from their label frequencies. It is primarily designed for comparing two human annotators; it is sometimes also reported for model predictions versus reference labels, but it should not replace task specific model metrics. Formula κ = (pₒ − pₑ) / (1 − pₑ) pₒ = observed agreement: the proportion of examples assigned the same label by both raters. pₑ = expected chance agreement: for each class, multiply the fraction assigned to it by rater A and rater B, and sum across classes. Beginner friendly step by step example Two reviewers independently label 10 support tickets as Urgent or Routine : Both say Urgent for 4 tickets. Reviewer A says Urgent , B says Routine for 2 tickets. Reviewer A says Routine , B says Urgent for 1 ticket. Both say Routine for 3 tickets. Step 1 — Observed agreement: They agree on 4 + 3 = 7 tickets , so pₒ = 7/10 = 0.70 (70%) . Step 2 — Each reviewer's label proportions: Reviewer A: Urgent = 6/10 = 0.60 ; Routine = 4/10 = 0.40 . Reviewer B: Urgent = 5/10 = 0.50 ; Routine = 5/10 = 0.50 . Step 3 — Expected chance agreement: pₑ = (0.60 × 0.50) + (0.40 × 0.50) = 0.30 + 0.20 = 0.50 . Step 4 — Kappa: κ = (0.70 − 0.50) / (1 − 0.50) = 0.20 / 0.50 = 0.40 . Interpretation: The reviewers agree on 70% of tickets, but their agreement above that expected from their marginal label frequencies corresponds to κ = 0.40 . It is not correct to call this “40% accuracy.” How does kappa compare with accuracy? Accuracy / raw agreement counts all matching assignments equally without adjusting for class prevalence. Kappa adjusts for the chance agreement estimated from how often each reviewer uses each class. κ = 1 means perfect agreement; κ = 0 means observed agreement matches estimated chance agreement; κ < 0 means less agreement than expected. Values are bounded by −1 and 1, though the achievable minimum depends on label marginals. Do not interpret a single κ value with rigid universal quality cutoffs: context and consequences matter. Python example Applications and limitations Label quality: assess consistency between two annotators before training on the labeled dataset. Classification analysis: useful as a supplementary agreement statistic, particularly when label frequencies are uneven. Prevalence sensitivity: heavily imbalanced labels can produce a surprisingly low kappa despite high raw agreement (the kappa paradox ). Report label frequencies and confusion patterns alongside it. Not a complete performance metric: it does not distinguish the practical costs of different errors or indicate which class is missed; inspect class specific precision/recall if assessing a classifier. Two raters only: ordinary Cohen's kappa compares two labelers on the same cases. For ordered categories, weighted kappa can assign differing penalties to near versus far disagreements; for more than two raters, use a suitable alternative. Undefined edge case: if pₑ = 1, the formula divides by zero; the statistic is undefined. Key takeaway: Kappa asks, “How much more do these two labelers agree than would be expected from their individual class label frequencies?” References scikit learn cohen kappa score Cohen (1960), A coefficient of agreement for nominal scales
What is Matthews correlation coefficient (MCC) in classification?
Matthews correlation coefficient (MCC) summarizes how well predicted and actual class labels agree, taking all four confusion matrix counts into account. For binary classification it is also called the phi coefficient . It is often informative when one class is much rarer than the other. Binary MCC formula MCC = (TP × TN − FP × FN) / √[(TP + FP)(TP + FN)(TN + FP)(TN + FN)]. Beginner friendly numerical example A model classifies 20 transactions as Fraud or Not Fraud: TP = 6: fraudulent transactions correctly flagged. TN = 8: legitimate transactions correctly cleared. FP = 2: legitimate transactions wrongly flagged. FN = 4: fraudulent transactions missed. Step 1 — Numerator: (6 × 8) − (2 × 4) = 48 − 8 = 40 . Step 2 — Denominator: √[(6 + 2)(6 + 4)(8 + 2)(8 + 4)] = √(8 × 10 × 10 × 12) = √9600 ≈ 97.98 . Step 3 — MCC: 40 / 97.98 ≈ 0.408 . This indicates positive but imperfect association between predicted and actual labels. Interpreting the score +1: perfect binary classification. 0: no correlation between predictions and true labels; it does not prove a model makes random predictions. −1: completely inverted binary predictions. Between: stronger positive scores generally indicate better overall label association, but consider class frequencies and operational consequences. Why MCC rather than only accuracy or F1? Here accuracy = (6 + 8)/20 = 70% and F1 = 2×6/(2×6 + 2 + 4) ≈ 66.7% , while MCC ≈ 0.408 . Accuracy counts correct cases, but a model predicting only the majority class can look good when classes are imbalanced. Positive class F1 combines TP, FP, and FN but ignores TN ; binary MCC includes all four counts. MCC is useful as an additional summary, not a replacement for per class recall, precision, and the cost of missing important cases. Python example Multiclass extension The same scikit learn function supports single label multiclass classification, using the complete K class confusion matrix rather than independently averaging several binary MCC scores. The multiclass result still has maximum +1 ; the most negative attainable value can depend on the number of classes and their distributions. Do not assume −1 is attainable in every multiclass problem. Use separate per class diagnostics to identify which categories are confused. Limitations and practical use Degenerate predictions: the binary denominator can be zero if a class is absent from actual or predicted labels. The formula is then undefined; scikit learn's implementation returns 0 in this situation. Interpret that output carefully. Rare positives: small test sets make any single metric unstable. Report class support and uncertainty alongside MCC. No business cost weighting: MCC does not know that missing a fraud may be more costly than a false alert. Consider cost sensitive metrics separately. Use cases: fraud detection, quality inspection, clinical classifiers, and other binary or multiclass tasks where class frequencies may be unequal. Key takeaway: MCC assesses the association between predictions and ground truth using the whole confusion matrix, making it a valuable complement to accuracy and F1, particularly on imbalanced data. References scikit learn matthews corrcoef scikit learn model evaluation
What is weighted Cohen's kappa, and when should you use it?
Weighted Cohen's kappa measures agreement between two raters who assign ordered categories (for example, severity 1 = low, 2 = medium, 3 = high). Unlike ordinary kappa, it penalizes a one step disagreement less than a two step disagreement. It corrects weighted observed disagreement for the disagreement expected from the raters' label frequencies. Linear versus quadratic disagreement weights For ordered ratings i and j: Linear weights: distance |i − j|; moving two categories apart costs twice as much as moving one. Quadratic weights: distance (i − j)²; moving two categories apart costs four times as much as moving one. These are disagreement penalties (zero means agreement). Scaling all weights by the same positive constant does not change weighted kappa. Worked example: six product quality ratings Two reviewers each assign scores 1, 2, or 3. Their paired ratings are: Product 1: 1 / 1 — agreement. Product 2: 1 / 2 — one step disagreement. Product 3: 2 / 2 — agreement. Product 4: 2 / 3 — one step disagreement. Product 5: 3 / 2 — one step disagreement. Product 6: 3 / 3 — agreement. Step 1 — Count observed disagreements. Three one step disagreements and no two step disagreements. Average linear penalty = (1+1+1)/6 = 0.5 ; average quadratic penalty = (1²+1²+1²)/6 = 0.5 . Step 2 — Determine rating proportions. Reviewer A uses ratings 1, 2, 3 two times each (1/3 each). Reviewer B uses them 1, 3, 2 times , respectively (1/6, 3/6, 2/6). Step 3 — Calculate disagreement expected from these frequencies. For every possible A/B rating pair, multiply its disagreement penalty by the product of the two rating proportions and sum. Expected linear penalty = 5/6 ≈ 0.8333 ; expected quadratic penalty = 7/6 ≈ 1.1667 . Step 4 — Adjust for expected disagreement. Weighted κ = 1 − observed weighted disagreement / expected weighted disagreement. Linear κ = 1 − 0.5/(5/6) = 0.40. Quadratic κ = 1 − 0.5/(7/6) ≈ 0.571. Quadratic kappa is higher in this example because both reviewers make only adjacent category errors, which receive a smaller penalty relative to the expected disagreement under squared distances. Quadratic kappa is not always higher than linear kappa. Python example When should you use it? When comparing two human annotators who rate severity, satisfaction, quality, or disease stage using an ordered scale. Linear is suitable when adjacent rating errors have approximately proportional costs; quadratic emphasizes larger disagreements more strongly. It can supplement evaluation of a model against ordinal reference labels, but label agreement and real world error costs still need separate examination. Limitations Order matters: do not use ordinal distance weights for unordered labels such as Cat, Dog, Bird. Spacing is assumed: codes 1, 2, 3 imply equally spaced severity steps even if practical differences are not equal. Prevalence sensitivity: score depends on both reviewers' category frequencies; report disagreement counts too. Small samples: estimates can be unstable, and if expected weighted disagreement is zero, kappa is undefined. Not multi rater by default: Cohen's weighted kappa compares two raters on the same items. Key takeaway: Weighted kappa asks whether two raters agree on an ordered scale , while treating near misses as less serious than distant disagreements. References scikit learn cohen kappa score Weighted kappa for ordinal scales, Journal of Classification
What is the Brier score in classification?
The Brier score measures the quality of a classifier's predicted probabilities by averaging squared differences between its probabilities and the actual outcomes. Lower is better: a score of 0 means all predictions assign probability 1 to the true outcome. Binary formula For N examples, with actual label yᵢ ∈ {0,1} and predicted probability pᵢ of class 1: Brier score = (1/N) × Σ(pᵢ − yᵢ)² Beginner friendly step by step example A model predicts whether each of four emails is spam (1) or not spam (0): Email 1: predicted spam probability 0.80 , actual 1 → (0.80 − 1)² = 0.04 . Email 2: probability 0.60 , actual 0 → (0.60 − 0)² = 0.36 . Email 3: probability 0.30 , actual 1 → (0.30 − 1)² = 0.49 . Email 4: probability 0.10 , actual 0 → (0.10 − 0)² = 0.01 . Step 1: Add squared errors: 0.04 + 0.36 + 0.49 + 0.01 = 0.90 . Step 2: Divide by four: Brier score = 0.90/4 = 0.225 . Interpretation: The confidently missed spam email (Email 3) incurs a larger penalty than the correct high confidence predictions. The number 0.225 is a loss , not an accuracy percentage: compare it with an appropriate baseline and other models on the same evaluation set. Why use probability scores rather than just accuracy? Two models can choose the same class labels but report very different probabilities. Brier score evaluates how sensible those probabilities are, not just whether they cross a decision threshold. Because errors are squared, a confidently wrong probability is penalized heavily. The Brier score is a strictly proper scoring rule : in expectation, predicting the true conditional probabilities minimizes the loss. Calibration versus discrimination Calibration: When a model predicts 0.7 across many similar cases, does the event occur roughly 70% of the time? Discrimination: Does the model tend to assign higher risk scores to events than to non events? The Brier score reflects both calibration and resolution/discrimination , so one good score does not independently prove that the model is calibrated or that its ranking is excellent. Inspect reliability diagrams and a ranking metric such as ROC AUC when those properties matter. Python example Multiclass extension For mutually exclusive classes, use each example's full predicted probability vector and a one hot true label: Multiclass Brier = (1/N) × Σᵢ Σ꜀ (pᵢ꜀ − yᵢ꜀)². For example, if the true class is Cat and predicted probabilities for [Cat, Dog, Bird] are [0.7, 0.2, 0.1], the example contributes (0.7 − 1)² + (0.2 − 0)² + (0.1 − 0)² = 0.14 . Scaling matters: the conventional binary Brier score ranges from 0 to 1; the unscaled multiclass version ranges from 0 to 2. In modern scikit learn, scale by half='auto' uses binary halving but leaves multiclass scores unscaled. Compare scores only when definitions and settings agree. Applications and limitations Useful for: risk prediction, churn probabilities, spam filtering, and comparing probabilistic classifiers. Limitations: its numerical value depends on outcome prevalence and evaluation population; compare to a prevalence based reference forecast. A single average can hide failure on important subgroups. Unequal error costs: Brier loss does not itself encode costs of false positives versus false negatives; decision thresholds and business evaluation are separate concerns. Do not confuse it with log loss: both evaluate probabilities, but log loss uses logarithmic penalties; Brier uses squared error. Key takeaway: Brier score answers “How close were the model's predicted probabilities to what actually happened?”—not simply “How often was its predicted class right?” References scikit learn brier score loss scikit learn model evaluation: Brier score
What is a calibration curve (reliability diagram) in classification?
A calibration curve (reliability diagram) checks whether predicted probabilities match observed event frequencies. It groups predictions into probability bins and plots mean predicted probability (x axis) against observed positive rate (y axis) . This lesson focuses on constructing and interpreting the chart , rather than re explaining calibration or probability losses. Step by step numerical example Suppose a binary model predicts whether a customer will renew a subscription. We have 12 held out predictions , grouped into three illustrative probability ranges: Low bin [0, 0.33): predicted probabilities 0.10, 0.20, 0.20, 0.30 ; observed outcomes 0, 0, 1, 0 . Mean prediction = 0.20 , observed positive rate = 1/4 = 0.25 . Plot (0.20, 0.25) . Middle bin [0.33, 0.67): predicted probabilities 0.40, 0.50, 0.50, 0.60 ; outcomes 0, 1, 0, 1 . Mean prediction = 0.50 , positive rate = 2/4 = 0.50 . Plot (0.50, 0.50) . High bin [0.67, 1]: predicted probabilities 0.70, 0.80, 0.80, 0.90 ; outcomes 1, 0, 1, 0 . Mean prediction = 0.80 , positive rate = 2/4 = 0.50 . Plot (0.80, 0.50) . How to read the diagram 1. Draw a diagonal reference line y = x , representing perfect agreement between predicted probability and observed frequency. 2. Plot each bin's average predicted probability against its actual positive fraction. 3. Point above the diagonal: events happen more frequently than predicted → the model underestimates positive event probability in that bin. 4. Point below the diagonal: events happen less frequently than predicted → the model overestimates positive event probability in that bin. 5. Near the line: the bin is approximately calibrated, but small bin counts can produce misleadingly reassuring points. In the example, the low bin is slightly above the line ( 0.25 vs 0.20 ), the middle bin is on it ( 0.50 vs 0.50 ), and the high bin is below it ( 0.50 vs 0.80 ): the model is overly optimistic about renewal in the high score group. For clarity, always specify which event is considered positive; terms like “overconfident” can otherwise be ambiguous. Python: create a reliability diagram The display uses the mean predicted probability per nonempty bin and observed positive frequency. Results may vary slightly from manually chosen bin boundaries when values land on an edge; the example intentionally avoids the 1/3 and 2/3 boundaries. Binning choices Uniform width bins (strategy='uniform'): same probability interval width; intuitive x axis ranges, but may leave very few samples in some bins. Quantile bins (strategy='quantile'): approximately equal sample counts; helpful for uneven score distributions, but bin widths may vary. Number of bins: many bins reveal local patterns but produce noisy rates; fewer bins are steadier but can hide local miscalibration. Show a histogram or counts per bin so viewers can judge how much data supports each point. Empty bins are omitted from scikit learn's returned curve. Applications and limitations Uses: inspect renewal, churn, fraud risk, and medical risk probability estimates before applying decision rules. Calibration is not ranking: a visually near diagonal plot does not show how effectively the model separates positive and negative examples. Small sample noise: the example's four cases per bin are only for learning; operational conclusions need many representative held out predictions and uncertainty checks. Population shift: a curve measured on old data may not describe current behavior. Multiclass: scikit learn's basic calibration curve expects binary labels; inspect multiclass probability calibration using suitable class wise one vs rest or multiclass specific methods. Key takeaway: Every reliability diagram point answers: “When this model predicted roughly x% , did the event actually occur about x% of the time?” References scikit learn calibration curve scikit learn CalibrationDisplay
What is Expected Calibration Error (ECE) in classification?
Expected Calibration Error (ECE) summarizes how closely a classifier's stated confidence matches its observed correctness. It splits predictions into confidence bins and averages the absolute confidence versus accuracy gaps, weighting each bin by its number of examples. Lower ECE is better , but zero estimated ECE does not guarantee perfect calibration. Formula ECE = Σ over bins [ (number of examples in bin / total examples) × |bin accuracy − mean bin confidence| ]. For standard top label ECE , confidence is the highest predicted class probability, and accuracy means whether that highest scoring class was correct. Beginner friendly numerical example Suppose we have 10 predictions , grouped into three nonempty confidence bins: Bin A: two predictions , each with 55% confidence; 1/2 correct = 50% accuracy . Gap = |0.50 − 0.55| = 0.05 ; weight = 2/10 = 0.20 ; contribution = 0.010 . Bin B: three predictions , each with 75% confidence; 2/3 correct ≈ 66.7% accuracy . Gap = |2/3 − 0.75| ≈ 0.0833 ; weight = 3/10 = 0.30 ; contribution = 0.025 . Bin C: five predictions , each with 90% confidence; 4/5 correct = 80% accuracy . Gap = |0.80 − 0.90| = 0.10 ; weight = 5/10 = 0.50 ; contribution = 0.050 . Add all weighted gaps: ECE = 0.010 + 0.025 + 0.050 = 0.085 , or 8.5 percentage points of average absolute bin level calibration gap. The value 8.5% is not classification error or accuracy . A small ECE can coexist with poor prediction accuracy, and opposite local deviations can be hidden by broad bins. Python: calculate top label ECE Binning choices and interpretation Equal width bins have fixed probability intervals; equal frequency (quantile) bins aim for similar sample counts. Too few bins can hide local miscalibration; too many bins create unstable observed accuracies. State the bin boundaries, empty bin handling, sample size, and class distribution. Different choices can produce different ECE values even for the same model. Top label vs classwise ECE Top label ECE considers only the predicted winning label's confidence and whether that prediction was correct. Classwise ECE assesses each class's predicted probability against whether the true label belongs to that class (one vs rest), then aggregates across classes. Top label calibration does not establish calibration of every class probability; the two definitions can disagree, and their numerical values should be identified explicitly. Limitations and applications Finite sample bias and instability: ECE is a binned estimate, not an exact population calibration error. Report the number of samples and consider sensitivity analyses or uncertainty intervals. Cancellation inside bins: over and underconfidence can cancel before taking the absolute gap, making ECE deceptively low. Rare classes: aggregate top label ECE can mask failure on minority classes; inspect classwise and subgroup calibration. Not a discrimination metric: ECE does not measure whether positive examples are ranked ahead of negatives. Pair it with suitable performance metrics. Applications: monitoring confidence quality for image classifiers, risk classifiers, and human review workflows where confidence affects decisions. Key takeaway: ECE is the sample weighted average of absolute gaps between mean confidence and observed accuracy in confidence bins . Always report which ECE variant and binning procedure you used. References Survey of uncertainty in deep neural networks Expected Calibration Error definition and classwise variants
What is Maximum Calibration Error (MCE) in classification?
Maximum Calibration Error (MCE) measures the largest absolute difference between average predicted confidence and observed accuracy among nonempty confidence bins. It helps locate the worst observed bin level mismatch, rather than the average mismatch. Formula MCE = max over nonempty bins |bin accuracy − mean bin confidence|. For standard top label MCE, confidence is the probability assigned to the predicted class, and accuracy is the fraction of predictions that are correct in that bin. A smaller MCE is better; 0 means all examined bins have matching observed accuracy and average confidence, not necessarily that the model is perfectly calibrated everywhere. Beginner friendly numerical example Consider 10 held out predictions divided into three confidence bins: Bin A (5 predictions): mean confidence 60% , 3 of 5 correct → accuracy 60% ; absolute gap = |0.60 − 0.60| = 0.00 . Bin B (3 predictions): mean confidence 75% , 2 of 3 correct → accuracy 66.7% ; absolute gap = |0.75 − 2/3| ≈ 0.0833 . Bin C (2 predictions): mean confidence 90% , 1 of 2 correct → accuracy 50% ; absolute gap = |0.90 − 0.50| = 0.40 . Step 1: Calculate each bin's absolute gap: 0.00, 0.0833, 0.40 . Step 2: Select the largest: MCE = max(0, 0.0833, 0.40) = 0.40 , or 40 percentage points . Step 3: Locate the problem: Bin C appears severely overconfident —its mean confidence is 90%, but only half of its two predictions are correct. Because it contains just two observations, do not treat this estimate as a reliable population level worst case guarantee. MCE versus ECE MCE: only the largest bin gap matters, irrespective of that bin's size. ECE: the sample count weighted average of absolute bin gaps matters. For these same ten predictions, ECE = (5/10 × 0) + (3/10 × 0.0833) + (2/10 × 0.40) = 0.105 , or 10.5 percentage points , whereas MCE = 40 percentage points . Both are estimates based on binning; they answer different questions: “worst observed confidence region?” versus “average observed discrepancy?” Python example Binning sensitivity and practical use Few bins: large bins can average away overconfidence and underconfidence, hiding local problems. Many bins: small counts make accuracy estimates volatile; one incorrect prediction may produce an outsized MCE. Uniform versus quantile bins: changing the edges or using approximately equal count bins may change which region has the largest gap. Inspect sample counts: report the worst bin's support and actual gap, not just the final MCE. Applications: risk assessment, human review systems, and selective classification where a badly calibrated confidence range can be consequential. Limitations Not a true worst case bound: a finite sample binned MCE is only the maximum gap among the observed bins , not proof that every possible confidence value is within that gap. No direction in the score: absolute values hide whether the model overestimates or underestimates; inspect each signed gap. Not an accuracy or ranking measure: also evaluate discrimination, error costs, and performance on important subgroups. Variant matters: top label, binary positive class, and classwise calibration may have different MCE definitions; state which one is used. Key takeaway: MCE finds the largest observed bin level calibration discrepancy , but its reliability depends heavily on how many examples each bin contains. References Guo et al., On Calibration of Modern Neural Networks Classifier calibration: a survey on how to assess and improve predicted class probabilities
What is Adaptive Calibration Error (ACE) in classification?
Adaptive Calibration Error (ACE) measures how far predicted probabilities differ from observed outcome frequencies, using equal frequency (approximately equal count) bins instead of fixed width probability intervals. The original ACE proposed by Nixon et al. (2019) evaluates each class's predicted probabilities , not only the winning class. Beginner friendly numerical example: adaptive bins Imagine six binary event probabilities, with true outcomes 0 (no event) or 1 (event): Example 1: probability 0.10, outcome 0. Example 2: probability 0.20, outcome 0. Example 3: probability 0.30, outcome 1. Example 4: probability 0.70, outcome 1. Example 5: probability 0.80, outcome 1. Example 6: probability 0.90, outcome 1. Step 1 — Sort scores. They are already in ascending order. Choose two equal frequency bins , each containing three examples. Step 2 — First bin (0.10, 0.20, 0.30): Mean predicted probability = (0.10 + 0.20 + 0.30)/3 = 0.20 . Observed positive rate = 1/3 ≈ 0.333 . Absolute calibration gap = |0.333 − 0.20| ≈ 0.133 . Step 3 — Second bin (0.70, 0.80, 0.90): Mean probability = (0.70 + 0.80 + 0.90)/3 = 0.80 . Observed positive rate = 3/3 = 1.00 . Absolute gap = |1.00 − 0.80| = 0.20 . Step 4 — Average the two gaps. Adaptive bin error = (0.1333 + 0.20)/2 ≈ 0.1667 , or 16.7 percentage points . This is a binary positive class adaptive binning illustration , not a claim that all definitions of multiclass ACE return this value. ACE formula and variants For K classes and R adaptive bins per class , the original classwise ACE is: ACE = (1 / (K × R)) × Σ over classes k and bins r |observed class k frequency − mean predicted class k probability|. Binary positive class adaptive ECE: apply equal frequency binning only to the positive class probabilities and average bin gaps; with equal counts it gives the worked example above. Top label adaptive ECE / ACE variant: sort each example's largest predicted class probability, and compare bin confidence against top label correctness. Original multiclass ACE: for each class, separately sort all probabilities assigned to that class; bin and compare them with class indicator outcomes, then average across classes and bins. These variants answer slightly different questions and should not be treated as interchangeable . Explicitly name the variant in reported results. Fixed width versus adaptive bins Fixed width binning: bins have predefined ranges (for example 0–0.2, 0.2–0.4, and so on), which can be empty or sparsely populated when scores cluster. Adaptive (quantile/equal frequency) binning: bin boundaries follow the score distribution so groups contain similar numbers of observations. Adaptive bins can make estimates more stable in high density regions, but their probability widths differ and they can hide calibration variation in broad sparse regions. Python example: binary positive class adaptive binned error Applications and limitations Useful for auditing probability outputs for image recognition, risk prediction, and other confidence sensitive classifiers. Number of bins matters: results change with bin count, ties, small datasets, and implementation choices. Rare classes and tiny probabilities: original multiclass ACE may be dominated by many near zero class probabilities; thresholded adaptive calibration error (TACE) was proposed to address this issue. Averages hide localized failures: inspect reliability plots and class/subgroup behavior, not just the final number. No accuracy guarantee: a low ACE does not imply good discrimination or useful decisions. Key takeaway: ACE adapts bins to the distribution of predicted probabilities . When using the term ACE, specify whether it is original classwise ACE, a top label adaptive variant, or binary positive class adaptive calibration error. References Nixon et al. (2019), Measuring Calibration in Deep Learning TorchUncertainty AdaptiveCalibrationError documentation