Machine Learning

Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.

How do you interpret a normalized confusion matrix?

A normalized confusion matrix converts raw classification counts into proportions , making error patterns easier to compare when classes have different numbers of examples. The meaning of each proportion depends on the denominator. Three ways to normalize Row wise (normalize='true'): divide each cell by its actual class row total . Rows sum to 1 when they have examples. This answers: “Of all actual Cats, what fraction did we predict as Dogs?” The diagonal is per class recall . Column wise (normalize='pred'): divide each cell by its predicted class column total . Columns with predictions sum to 1. This answers: “Of all predicted Dogs, what fraction are really Dogs?” The diagonal is per class precision . Global (normalize='all'): divide every cell by the total number of samples . All cells together sum to 1 (provided there are samples). It shows each outcome's proportion of the entire dataset; diagonal values sum to accuracy . Numerical example (no table) Imagine 12 images classified as Cat, Dog, or Bird: Actual Cat (5): predicted Cat 4 , Dog 1 , Bird 0 . Actual Dog (4): predicted Cat 1 , Dog 2 , Bird 1 . Actual Bird (3): predicted Cat 0 , Dog 1 , Bird 2 . Focus on the cell actual Dog → predicted Bird , which contains 1 : Row wise: Dog row total = 4, so 1 / 4 = 25% of actual dogs were mistaken for birds. Column wise: Bird prediction column total = 3 (0 Cats + 1 Dog + 2 Birds), so 1 / 3 ≈ 33.3% of predicted birds were actually dogs. Global: 12 total images, so 1 / 12 ≈ 8.3% of all images were dogs predicted as birds. The raw count stays 1; only its denominator changes. Python example Which should you use? Row wise: when investigating missed actual classes or comparing recall across classes with unequal support. Column wise: when investigating how trustworthy predictions of each class are. Global: when measuring how much each type of prediction or error contributes to overall results. Limitations: A normalized cell hides its raw sample count: 100% from one example is far less reliable than 100% from 1,000. Always examine counts and class support. Rows or columns with zero examples/predictions cannot be interpreted as meaningful recall/precision. Takeaway: Ask “out of what?” before interpreting any percentage in a normalized confusion matrix. Reference: scikit learn confusion matrix documentation.

How do you calculate per-class precision, recall, and F1-score in multiclass classification?

Per class metrics evaluate each label separately by temporarily treating that label as the positive class and every other label as “not that class.” This is called one vs rest evaluation ; it does not require retraining the classifier. Step by step numerical example Suppose a model classifies 12 photos as Cat, Dog, or Bird. Read each line as actual label → predicted counts : Actual Cat (5): Cat 4, Dog 1, Bird 0. Actual Dog (4): Cat 1, Dog 2, Bird 1. Actual Bird (3): Cat 0, Dog 1, Bird 2. Let's calculate the scores for Dog : 1. TP = 2: two actual dogs were predicted as Dog. 2. FP = 2: one cat and one bird were incorrectly predicted as Dog. 3. FN = 2: one dog was predicted as Cat and another as Bird. 4. TN = 6: remaining photos were neither actual Dog nor predicted Dog. TN is useful for the one vs rest view but is not needed in these three formulas. 5. Dog precision = TP / (TP + FP) = 2 / 4 = 50%. 6. Dog recall = TP / (TP + FN) = 2 / 4 = 50%. 7. Dog F1 = 2 × precision × recall / (precision + recall) = 50%. Now repeat the same calculation, changing only which label counts as positive: Cat: TP 4, FP 1, FN 1 → precision 80% , recall 80% , F1 80% . Bird: TP 2, FP 1, FN 1 → precision 66.7% , recall 66.7% , F1 66.7% . Python example Interpretation Dog's 50% precision means half of Dog predictions are correct ; its 50% recall means half of real dogs are found . Per class scores reveal exactly where errors occur; a single aggregate number can obscure that. For zero predicted examples , precision is undefined; for zero actual examples , recall is undefined. In this example the code explicitly uses zero division=0 so such scores are reported as zero. Inspect support before drawing conclusions. Key takeaway: In multiclass evaluation, pick a class, count TP/FP/FN against all other classes, calculate its three scores, and repeat for every label. Reference: scikit learn: precision recall fscore support.

How do you evaluate a multiclass classifier when classes are imbalanced?

When some classes are much rarer than others, overall accuracy can look excellent even when the model fails on the classes that matter most. Evaluate each class and choose a headline metric based on the cost of the mistakes. A beginner friendly example Suppose 100 support tickets have three actual labels: Routine: 90 tickets Urgent: 8 tickets Critical: 2 tickets A lazy model predicts Routine for all 100 : Accuracy = 90 / 100 = 90% . That sounds good, but the model misses every Urgent and Critical case. Recall: Routine = 100% , Urgent = 0% , Critical = 0% . Balanced accuracy = (100% + 0% + 0%) / 3 ≈ 33.3%. Macro F1 ≈ 31.6% : Routine F1 = 180/190 ≈ 94.7%, while the other two class F1 scores are zero; average ≈ 31.6%. Weighted F1 ≈ 85.3% , which still masks the rare class failure. The lesson is not to ignore accuracy, but to avoid using it alone . Practical evaluation checklist 1. Inspect class support: report actual counts for each class, including rare labels. 2. Start with a majority class baseline: ask whether the new model improves meaningfully beyond always predicting the most common label. 3. Look at per class recall and precision: if missing a Critical ticket is costly, prioritize Critical recall; also watch false alarms and the precision of Critical predictions. 4. Select an appropriate overall metric: balanced accuracy or macro F1 can expose poor performance across classes. Weighted F1 can supplement these, but should not replace rare class reporting. 5. Inspect confusion patterns: identify which classes are being mistaken for the majority class. 6. Evaluate on representative held out data: keep train/test records separate, and stratify splits where feasible; with only two Critical examples, measured performance is highly unstable. 7. Show uncertainty: rare class scores based on a handful of examples can swing dramatically. Gather more representative evaluation examples where possible, or use carefully designed cross validation and uncertainty intervals. 8. Use metrics tied to the operational cost: if missing a Critical case costs far more than an extra review, measure its recall and the downstream review burden rather than optimizing a generic overall score. Python example Common pitfalls Do not rebalance the test set just to make a score look better. Evaluate against realistic deployment prevalence; a separate balanced diagnostic set may also help. Do not confuse resampling with evaluation. Oversampling or SMOTE, if used for training, belongs inside training folds, never before splitting the data. No single metric solves every imbalance problem. For rare, high stakes classes, examine precision recall behavior and application specific costs. Key takeaway: For imbalanced multiclass classification, report per class performance, support, balanced accuracy or macro F1, and the real world cost of mistakes —not accuracy alone. References scikit learn model evaluation scikit learn classification report

What is balanced accuracy, and when should you use it?

Balanced accuracy measures how well a classifier recognizes each actual class, giving every class equal weight regardless of its frequency. In scikit learn, it is the arithmetic mean of class wise recall. Formula Recall for a class = correctly predicted examples of that class / total actual examples of that class. Balanced accuracy = sum of recall values for all observed classes / number of observed classes. Standard accuracy = number of correct predictions / total number of predictions. Beginner friendly worked example Suppose there are 14 support tickets labeled Routine, Urgent, or Critical: Routine: 8 actual; 7 correctly predicted → recall = 7/8 = 87.5% . Urgent: 4 actual; 2 correctly predicted → recall = 2/4 = 50% . Critical: 2 actual; 1 correctly predicted → recall = 1/2 = 50% . Step 1 — Standard accuracy: (7 + 2 + 1) / 14 = 10/14 ≈ 71.4% . Step 2 — Balanced accuracy: (87.5% + 50% + 50%) / 3 = 62.5% . Why are they different? Routine represents most examples, so standard accuracy counts its many correct predictions more heavily. Balanced accuracy treats the recall of Routine, Urgent, and Critical equally. The lower score highlights weaker recognition of the smaller classes. When is it useful? When class sizes differ greatly and you want a simple score that does not let the majority class dominate. When missing an instance from any class deserves attention and class wise recall is relevant. When comparing classifiers on the same representative held out data , alongside actual class counts. What balanced accuracy does not tell you It does not measure precision or directly penalize false alarms for a given predicted class. Inspect precision when false alerts have operational costs. It can conceal one poorly recognized class if others score well; always inspect each class's recall. Estimates for a class with only a few test examples are unstable. For standard multiclass classification, with unadjusted scores, a model that always predicts one of K observed classes earns 1/K balanced accuracy. The adjusted=True option in scikit learn shifts chance level performance to 0. Python example Remember: Standard accuracy asks, “What fraction of all predictions were correct?” Balanced accuracy asks, “How well were the actual classes recognized on average?” Reference: scikit learn balanced accuracy score.

How do you evaluate a multilabel classification model?

Evaluate a multilabel classifier by checking both complete label sets and individual label decisions. Unlike multiclass classification, where each example belongs to one class, a multilabel example can have several correct labels simultaneously. Beginner friendly example Imagine three photos, each potentially labeled Dog, Outdoor, and Person . Every label is independently either present (1) or absent (0). Photo 1: actual {Dog, Outdoor}; predicted {Dog}. The prediction misses Outdoor. Photo 2: actual {Outdoor, Person}; predicted {Outdoor, Person}. Perfect match. Photo 3: actual {Dog}; predicted {Dog, Person}. The prediction wrongly adds Person. Each photo has three label decisions, giving 3 × 3 = 9 decisions in total. 1. Subset accuracy (exact match accuracy) A photo counts as correct only when its entire predicted label set exactly matches the actual set. Only Photo 2 is an exact match. Subset accuracy = 1/3 ≈ 33.3%. This is deliberately strict: one extra or missing label makes an entire photo incorrect. 2. Hamming loss Hamming loss measures the fraction of individual label decisions that are wrong. Photo 1 has one missing label; Photo 3 has one extra label; Photo 2 has zero errors. Hamming loss = 2/9 ≈ 22.2%. Lower is better; 0 means no label level mistakes. Hamming loss is more forgiving than subset accuracy because it counts errors per label, not per whole photo. 3. Multilabel F1 score F1 balances precision and recall. In this example, across all photo label decisions: True positives = 4 , false positives = 1 , false negatives = 1 . Micro precision = 4/5 = 80% . Micro recall = 4/5 = 80% . Micro F1 = 80%. For multilabel tasks, the choice of F1 averaging matters: Micro F1: pools label decisions across the dataset. Macro F1: averages the separate F1 of each label equally. Samples F1: calculates an F1 for each photo's predicted versus actual label sets and then averages across photos. In this example: (2/3 + 1 + 2/3)/3 ≈ 77.8% . Do not assume micro F1 equals subset accuracy in multilabel classification. Python example Choosing a metric Use subset accuracy when a complete set of labels must be perfect. Use Hamming loss when each missed or extra label matters independently. Use micro F1 for an overall label decision summary and macro/per label F1 to expose weak performance on uncommon labels. Always show per label support , evaluate on representative held out data, and inspect whether your score changes when prediction thresholds change. Watch out for examples with no positive labels, where some per sample precision/recall/F1 expressions require a defined zero division convention. Key takeaway: Multilabel evaluation is not one number. Exact match accuracy measures perfect sets; Hamming loss measures individual mistakes; F1 measures precision recall balance. References scikit learn model evaluation scikit learn hamming loss

What is Hamming loss in multilabel classification?

Hamming loss is the proportion of individual label decisions that a multilabel classifier gets wrong. Unlike an exact match measure, it gives partial credit when a prediction gets some labels right. Formula For N samples and L possible labels, each represented by a 0 (absent) or 1 (present): Hamming loss = number of mismatched label indicators / (N × L). A lower score is better: 0 means every label indicator is correct; 1 means all indicators are wrong. Step by step numerical example Imagine three movie descriptions, each with possible labels Action, Comedy, Romance . Compare actual and predicted binary indicators in this fixed label order: Movie 1: actual [1, 0, 1]; predicted [1, 1, 1]. Only Comedy differs → 1 error . Movie 2: actual [0, 1, 0]; predicted [0, 1, 0]. All indicators agree → 0 errors . Movie 3: actual [1, 1, 0]; predicted [0, 1, 1]. Action and Romance differ → 2 errors . Step 1: Count incorrect indicators: 1 + 0 + 2 = 3 . Step 2: Count all indicators: 3 movies × 3 labels = 9 . Step 3: Hamming loss = 3 / 9 = 0.333 (33.3%) . So one third of individual label decisions are wrong. Hamming loss vs. subset accuracy Hamming loss measures errors per label; a sample with just one incorrect label incurs only one label level error. Subset accuracy counts a sample as correct only if every label matches . Here only Movie 2 is fully correct: 1/3 = 33.3% subset accuracy . The equal percentages here are coincidental , not an equivalence: Hamming loss is minimized, while subset accuracy is maximized. Their denominators and meaning differ. Python example How to interpret and when to use it Useful when each missing or extra label has a comparable cost, and partial correctness matters. Report it alongside per label precision/recall and label support, especially for rare labels. Limitation: In sparse multilabel datasets, predicting all labels absent can produce a deceptively low Hamming loss because most indicators really are zero. Limitation: False positive and false negative indicators count equally, although their real world costs may differ. For unequal sample or label importance, use additional appropriate metrics; do not assume ordinary Hamming loss reflects those priorities. Key takeaway: Hamming loss answers, “Out of all individual label indicators, what fraction were wrong?” Reference: scikit learn hamming loss documentation.

What is subset accuracy in multilabel classification?

Subset accuracy (exact match accuracy) measures the fraction of examples for which the predicted set of labels matches the true set perfectly. If even one label is missing or incorrectly added, that whole example counts as incorrect. Formula Subset accuracy = number of samples with all labels correct / total number of samples. For multilabel data, compare each sample's full binary indicator vector (or label set), not individual positions in isolation. Beginner friendly step by step example Suppose an image tagging model assigns any combination of Animal, Outdoor, Person, and Vehicle : Image 1: actual {Animal, Outdoor}; predicted {Animal, Outdoor} → exact match . Image 2: actual {Person, Vehicle}; predicted {Person} → not a match , since Vehicle is missing. Image 3: actual {Animal}; predicted {Animal, Outdoor} → not a match , since Outdoor was added. Image 4: actual {Outdoor, Person}; predicted {Outdoor, Person} → exact match . Image 5: actual {Vehicle}; predicted {Animal} → not a match . Step 1: Count perfect label sets: Images 1 and 4 → 2 matches . Step 2: Count all images: 5 . Step 3: Subset accuracy = 2/5 = 0.40 (40%) . The other three images receive no partial credit, even if some of their tags are correct. Why does it become strict as label count grows? If each of L label decisions is independently correct with probability 0.9 , then under that simplified independence assumption, the chance of all labels being right is 0.9^L : With 2 labels : 0.9² = 81% . With 5 labels : 0.9⁵ ≈ 59% . With 10 labels : 0.9¹⁰ ≈ 35% . This is an illustration, not a universal formula : real label errors can be dependent, labels vary in difficulty, and label cardinality matters. Still, the example shows why exact match accuracy may decline as the number of possible labels increases. Python example When is it useful? When the entire set must be correct before an automated action can proceed, such as a rule based multi tag routing system. As a strict quality measure alongside less restrictive label level metrics. Limitations No partial credit: one incorrect label is treated the same as several incorrect labels for that example. Sensitive to label count: comparisons across tasks with different label vocabularies can be misleading. Can hide label specific problems: report per label metrics and class support to locate weak or rare labels. Empty label sets matter: correctly predicting no labels also counts as an exact match; inspect prevalence if empty sets dominate. Key takeaway: Subset accuracy asks, “How often did the model get the complete label set exactly right?” Reference: scikit learn accuracy score.

What is sample-averaged F1-score in multilabel classification?

Sample averaged F1 scores each example's complete predicted label set separately, then takes the arithmetic mean across examples. It is useful when the quality of each person's document, image, or other individual prediction matters. Formula For sample i, count TPᵢ (correctly predicted positive labels), FPᵢ (extra positive labels), and FNᵢ (missing positive labels). F1ᵢ = 2TPᵢ / (2TPᵢ + FPᵢ + FNᵢ) when the denominator is positive. Samples F1 = (F1₁ + ... + F1ₙ) / n . Beginner friendly example An image model can attach any of Cat, Outdoor, Person to each photo: Photo 1: actual {Cat, Outdoor}, predicted {Cat, Person}. TP=1, FP=1, FN=1, so F1=2/(2+1+1)= 0.50 . Photo 2: actual {Person}, predicted {Person}. TP=1, FP=0, FN=0, so F1=2/(2+0+0)= 1.00 . Photo 3: actual {Cat, Person}, predicted {Cat}. TP=1, FP=0, FN=1, so F1=2/(2+0+1)= 0.667 . Now average the three sample scores : (0.50+1.00+0.667)/3 ≈ 0.722 (72.2%) . What makes it different? Samples F1: calculates an F1 for each individual example, then averages; a photo with many labels does not automatically count more than another photo. Micro F1: pools label level TP, FP, and FN across all samples before calculating F1; examples with more positive label decisions can have greater influence. Macro F1: calculates F1 separately for each label, then averages over labels; each label has equal weight. For the example, pooled TP=3, FP=1, FN=2, giving micro F1=6/9≈66.7% , different from samples F1≈72.2%. Python Empty label edge cases If a sample has no actual positives and no predicted positives , TP=FP=FN=0; its F1 is mathematically undefined, not intrinsically perfect. scikit learn defaults to reporting 0 and raising an UndefinedMetricWarning; choose zero division=1 if your evaluation convention counts a correct empty set prediction as 1, and explain that choice. If actual positives exist but the model predicts none, F1 is 0 . If it predicts positives but none are actually present, F1 is also 0 . Compare systems using the same zero division rule, especially when many examples have no labels. Applications and limitations Good for tagging or recommendation tasks where the completeness and correctness of each example's label set matters. It can hide poor performance on rare labels; inspect per label scores and support. It does not account for unequal harm from different missing or extra labels. Takeaway: Samples F1 is an average of per example F1 scores , not an F1 computed from pooled counts. Reference: scikit learn f1 score.

How do you choose classification thresholds for multilabel models?

A multilabel classifier can predict several labels for the same example. It often outputs a separate score for each label; thresholds turn these scores into yes/no decisions. Use one shared threshold as a simple starting point, or tune separate thresholds when labels have different frequencies, score distributions, or error costs. Beginner friendly example: shared vs per label thresholds A photo receives model scores for Animal = 0.72, Outdoor = 0.43, Person = 0.28 . Shared threshold = 0.50: predict {Animal} , because only 0.72 reaches 0.50. Per label thresholds: Animal = 0.70 , Outdoor = 0.40 , Person = 0.30 . Predict {Animal, Outdoor} ; Person stays absent. If the actual labels are {Animal, Outdoor} , the per label thresholds get the full set right while the shared threshold misses Outdoor. One example does not prove the customized thresholds are better ; compare across held out examples. How to select thresholds without leakage 1. Split the dataset into training, validation, and untouched test partitions (respect groups or time order where needed). 2. Train the model only on training data. Use an initial shared threshold such as 0.50 as a baseline, not an automatic optimum. 3. Collect validation scores for every label. Try candidate thresholds (for example 0.20, 0.30, …, 0.80), and evaluate against actual validation labels. 4. Choose the objective beforehand: for example macro F1, samples F1, a minimum recall requirement for critical labels, or a cost sensitive utility. A label by label F1 optimization may not maximize an aggregate sample level objective. 5. Choose shared or per label thresholds. Separate thresholds are useful when label score distributions differ; consider shrinkage or a common threshold for labels with few positives to avoid overfitting. 6. Freeze thresholds before test evaluation. Report the final scores on the untouched test split and monitor them after deployment. Precision–recall tradeoff Lowering a label's threshold usually produces more positive predictions: it can recover missed positives (higher recall), but may increase false alarms and lower precision. Raising a label's threshold typically reduces alerts: it may improve precision but miss more positives and reduce recall. The exact movement of precision is not guaranteed to be monotonic for every finite dataset. Inspect the actual precision–recall curve. Small Python example Practical cautions Avoid leakage: selecting thresholds on test data makes the final test score optimistic. Repeated validation tuning can also overfit; use cross validation or a second validation set for large searches. Data imbalance: a rare label may have too few positive validation examples for a stable individually optimized threshold. Calibration: probabilities can be miscalibrated; check calibration when interpreting scores as probabilities or using probability based decisions. Changing prevalence and costs: re evaluate thresholds when data or business requirements shift. Correlated labels: choosing each threshold in isolation may ignore useful constraints between labels. Takeaway: Start with a shared threshold, tune on validation data for a clearly chosen objective, consider per label thresholds when supported by enough data, and test only after freezing all choices. References scikit learn precision recall curve scikit learn TunedThresholdClassifierCV — binary estimator threshold tuning; multilabel specific tuning requires applying the approach per label or implementing a suitable multilabel objective.

What is the Jaccard score in classification?

Jaccard score , also called the Jaccard index or intersection over union (IoU) , measures how much the predicted set of positive labels overlaps with the actual set. Formula Jaccard = |Actual labels ∩ Predicted labels| / |Actual labels ∪ Predicted labels| Equivalently, for a label or set of labels with a nonempty union: TP / (TP + FP + FN) . 1 means perfect overlap; 0 means no shared positive labels. Step by step multilabel example Suppose an image can have any of these tags: Dog, Outdoor, Person, Running . Actual labels: {Dog, Outdoor, Running}. Predicted labels: {Dog, Outdoor, Person}. 1. Intersection: {Dog, Outdoor} → 2 matching positive labels. 2. Union: {Dog, Outdoor, Running, Person} → 4 labels appearing in either set. 3. Jaccard = 2 / 4 = 0.50 (50%) . The model got two relevant tags right, added an incorrect tag, and missed one correct tag. Jaccard compared with F1 and subset accuracy Jaccard: gives partial credit for overlap, while penalizing missing and extra tags. F1 (for the same label set): here precision = recall = 2/3, so F1 = 2/3 ≈ 66.7% . For the same TP, FP, FN counts, F1 = 2J/(1+J) whenever defined. The two scores rank individual predictions consistently, but their dataset level averaged values need not obey this relationship. Subset accuracy: this image is incorrect (0) because the complete predicted set differs from the true set; it gives no partial credit. Jaccard averaging options average='samples': Jaccard per example, then average; often useful for multilabel tagging. average='micro': pool TP, FP, FN across labels and samples before computing Jaccard. average='macro': compute Jaccard separately for each label, then average labels equally. average='weighted': average label wise Jaccard scores weighted by true label support. average=None: return one score for each label rather than a single summary. Python example When is it useful? Image tagging and multilabel document classification: assess overlap between predicted and actual label sets. Segmentation: IoU compares predicted and true pixel sets for a class; handle background and absent classes deliberately. Information retrieval or recommender set comparisons: assess shared relevant items relative to the union. Limitations and edge cases Both sets empty: the union is zero, so the mathematical ratio is undefined. scikit learn defaults to 0 with a warning ; configure zero division explicitly when choosing another convention. Rare labels: micro or weighted averages can mask them; inspect per label results alongside the summary. Unequal error costs: ordinary Jaccard treats each extra/missing positive label symmetrically; a business critical tag may require separate evaluation. Small samples: an IoU score alone does not convey how many examples support it. Key takeaway: Jaccard measures the overlap divided by the union of predicted and true positive labels; it rewards partial correctness without accepting extra or missing tags as perfect. References scikit learn Jaccard score scikit learn model evaluation