Machine Learning

Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.

What is a precision-recall curve?

A precision–recall (PR) curve plots a classifier's precision against its recall as the decision threshold changes. It shows how many positive predictions are correct and how many actual positives the model finds at different operating points. What are its axes? X axis: Recall = TP / (TP + FN). Of actual positives, how many were found? Y axis: Precision = TP / (TP + FP). Of predicted positives, how many were correct? Each threshold gives a point (recall, precision) on the curve. A curve concentrated near the upper right generally indicates better positive class detection with fewer false alarms. Numerical example: fraud alerts A model scores six transactions : Fraud (positive): 0.90, 0.70, 0.40. Legitimate (negative): 0.80, 0.30, 0.10. Flag a transaction when its score is at least the threshold. Threshold 0.90: TP=1, FP=0, FN=2 → precision 100%, recall 33.3% . Threshold 0.80: TP=1, FP=1, FN=2 → precision 50%, recall 33.3% . Threshold 0.70: TP=2, FP=1, FN=1 → precision 66.7%, recall 66.7% . Threshold 0.40: TP=3, FP=1, FN=0 → precision 75%, recall 100% . Threshold 0.10: TP=3, FP=3, FN=0 → precision 50%, recall 100% . These points form a step like curve. Notice that precision can improve when recall rises (from threshold 0.80 to 0.70): precision is not guaranteed to decrease monotonically . Python example Pass continuous scores or positive class probabilities, rather than hard 0/1 predictions. In scikit learn, the returned precision and recall arrays each have one more element than thresholds; the additional point is the plotting endpoint at recall 0, precision 1. When is a PR curve useful? Rare positive problems: fraud, unusual defects, and event detection, where precision among alerts matters. Threshold selection: find an operating point with enough recall without too many false positive alerts. Model comparison: compare precision at recall levels important to the application instead of considering accuracy alone. How is it different from an ROC curve? PR curve: precision versus recall, focusing on the positive class and positive prediction quality. ROC curve: true positive rate versus false positive rate, considering correct positive detection relative to all actual negatives. PR curves often provide more actionable insight when positive examples are scarce. What about PR AUC and average precision? A PR curve can be summarized by area under the precision–recall curve (PR AUC) or average precision (AP) . They are related but not necessarily numerically identical: scikit learn's average precision score uses a recall weighted sum of precisions and does not interpolate linearly between the curve points. The next question covers when these summaries are more informative than ROC AUC. In short: a precision–recall curve shows the trade off between catching real positives and keeping positive alerts reliable across classification thresholds. References scikit learn: precision recall curve scikit learn: Precision–Recall example

When is PR-AUC more useful than ROC-AUC?

PR AUC is often more useful than ROC AUC when the positive class is rare and the quality of positive alerts matters. It summarizes precision versus recall across thresholds, emphasizing how many flagged examples are truly positive. Why can ROC AUC look good for rare events? ROC uses true positive rate (TPR) and false positive rate (FPR) . FPR divides false positives by all actual negatives . With thousands of negatives, even many false alerts can produce a small FPR. Precision divides true positives by all predicted positives , making false alerts immediately visible. Numerical example: fraud alerts A system evaluates 10,000 transactions, including 100 genuine frauds and 9,900 legitimate ones. At a chosen threshold: TP = 80; FN = 20. FP = 198; TN = 9,702. A 2% false positive rate sounds low, but 71.2% of flagged transactions are legitimate . The PR curve makes this operational problem obvious. These values describe one threshold , not an ROC AUC or PR AUC score: both AUC metrics require scores over many thresholds. Choosing the metric Prefer PR AUC / average precision for rare disease detection, fraud alerts, anomaly detection, and information retrieval when positive predictions should be useful. ROC AUC is useful for general positive–negative ranking performance and comparisons across false positive rates. Use both when practical , especially if the business cost of false positives and false negatives is important. Neither metric alone sets the final operating threshold. Compare within similar positive class prevalence. PR baselines depend strongly on the proportion of positives; a random ranker's expected precision baseline is approximately the positive prevalence. Python example Terminology matters: scikit learn's average precision (AP) is the recall weighted sum of precision at observed operating points; it need not equal trapezoidal PR AUC , which linearly interpolates between points. State which method is used when reporting results. In short: choose a PR based summary when rare positives and false alarm burden are central; use ROC AUC for broader ranking discrimination, and still inspect performance at the deployment threshold. References scikit learn: Precision–Recall evaluation scikit learn: Average precision

What is log loss in classification?

Log loss (logarithmic loss), also called cross entropy loss , evaluates how well a classifier's predicted probabilities agree with the true labels. It penalizes confidently wrong predictions much more heavily than uncertain ones. Binary classification formula For a true label y of 0 or 1 and a predicted probability p for class 1: For N examples, mean log loss averages the individual losses. If y = 1, loss is ln(p). If y = 0, loss is ln(1 p). Lower is better: zero means perfect probabilities for the observed labels. Log loss uses the natural logarithm . Beginner friendly numerical example A spam classifier predicts the probability that an email is spam. Case A: correct, confident prediction Actual: spam (y=1); predicted spam probability p=0.9. Loss = ln(0.9) ≈ 0.105. Case B: correct, but uncertain prediction Actual: spam (y=1); predicted probability p=0.6. Loss = ln(0.6) ≈ 0.511. Case C: confidently wrong prediction Actual: spam (y=1); predicted probability p=0.01. Loss = ln(0.01) ≈ 4.605. Why it matters: all three emails might be evaluated against the same actual label. A correct and confident prediction has a small loss; a confidently wrong prediction receives a much larger loss. Accuracy alone only checks the predicted class at a chosen threshold and does not capture this probability quality. Calculate the average for two emails Suppose two emails have: Email 1: y=1, p=0.9 → loss 0.10536. Email 2: y=0, p=0.2 → loss ln(0.8)=0.22314. Python example Pass probabilities , not predicted 0/1 class labels. For multiclass classification, cross entropy uses the negative log of the probability assigned to the true class, averaged over examples; the probability columns must match the class label order. When is log loss useful? Training probabilistic classifiers , such as logistic regression and neural network classifiers. Comparing predicted probability quality , not just classification accuracy. Flagging overconfident errors: extremely incorrect probabilities receive a large penalty. Important limitations A low log loss does not guarantee the best threshold dependent precision, recall, or business outcome. Perfectly predicted probability 0 for a true positive—or probability 1 for a true negative—has infinite theoretical loss. Numerical implementations clip extreme probabilities to finite bounds. Log loss is sensitive to unusually confident mistakes, so compare it on a representative held out set. In short: classification log loss measures the quality of predicted probabilities and strongly penalizes predictions that are confident but wrong . References scikit learn: log loss Google ML Crash Course: Log loss

What is probability calibration?

Probability calibration means checking whether a classifier's predicted probabilities match how often events actually occur. A model is well calibrated if, among examples assigned about 70% probability of the positive class, approximately 70% really are positive . Why does calibration matter? A classifier can rank cases correctly but still report misleading probabilities. For example, two models might identify the same patients as higher risk, while one consistently predicts 90% for cases that actually occur only 60% of the time. That model is overconfident . Overconfident: predicted probabilities are higher than observed event frequencies. Underconfident: predicted probabilities are lower than observed event frequencies. Well calibrated: predicted probabilities agree with observed frequencies, within sampling uncertainty. Calibration is not the same as accuracy or ROC AUC. Accuracy checks hard class labels; ROC AUC assesses ranking. Calibration checks whether the numeric probabilities are trustworthy. Beginner friendly numerical example A loan default model predicts approximately 0.80 default probability for 20 borrowers . Suppose 12 of those borrowers actually default. The model predicted an 80% event rate, but only 60% defaulted in this group—suggesting overconfidence by 20 percentage points . One small group cannot establish overall model calibration; assess multiple groups on enough held out data. How do you evaluate calibration? 1. Collect out of sample predicted probabilities and actual binary labels. 2. Group predictions into bins, such as around 0.1, 0.2, ..., 0.9. 3. For each bin, calculate the mean predicted probability and the observed positive fraction . 4. Plot these points in a reliability diagram (calibration curve) . Ideal calibration follows the diagonal observed frequency = predicted probability. Python example: reliability diagram This tiny example demonstrates the method , not a statistically reliable assessment; use more observations for real evaluation. Can you improve calibration? Yes. A separate calibration stage can map model scores to probabilities using approaches such as sigmoid/Platt scaling , isotonic regression , or temperature scaling . Fit any calibration method using data separate from the examples used to train the base model , or a suitable cross validation procedure, to avoid misleadingly optimistic results. Important: metrics such as Brier score and log loss assess probability quality but also reflect discrimination and uncertainty; lower scores do not by themselves prove better calibration. When does it matter most? Healthcare risk scores: people need to interpret predicted probabilities. Loan and fraud risk: thresholds may depend on estimated risk or expected costs. Forecasting and prioritization: downstream teams use probabilities to make decisions. In short: a calibrated 80% prediction means about 80 out of 100 comparable cases should experience the event—not that any individual event is guaranteed. Reference scikit learn: Probability calibration

What is a calibrated classifier?

A calibrated classifier is a classification model whose output probabilities are adjusted or learned so that they better match observed event frequencies. When it predicts about 70% risk for a sufficiently large group of similar cases, roughly 70% of those cases should experience the event . What makes it different from an ordinary classifier? A classifier may correctly rank examples from low risk to high risk while still giving misleading probability values. A calibrated classifier produces probabilities intended to be meaningful for risk based decisions , not just scores for ranking. Calibration is evaluated on held out data : training set confidence alone is not proof of calibration. Beginner friendly numerical example Suppose a credit risk classifier assigns a default probability of 0.80 to 50 borrowers in a validation group. Only 30 actually default: This suggests the model is overconfident in this group. A suitable calibration mapping may adjust such predictions from 0.80 toward 0.60 , assuming the pattern is supported by enough representative calibration data. The mapping is learned from many examples, not by manually subtracting 20 percentage points from every prediction. Calibrated outputs also are not guaranteed to equal observed outcomes in every small subgroup. How do you build one? 1. Fit a base classifier to training data. It might generate decision scores or uncalibrated probabilities. 2. Obtain out of sample predictions from the base classifier for calibration. 3. Fit a calibration mapping that converts the model's outputs into better probability estimates. 4. Evaluate the calibrated model on a separate test set , checking reliability diagrams, log loss or Brier score, and the task's decision metrics. Common calibration mappings include: Sigmoid (Platt scaling): fits a smooth sigmoid mapping; a useful default for limited calibration data. Isotonic regression: learns a flexible nondecreasing mapping but can overfit with small calibration datasets. Temperature scaling: adjusts logits before converting them to probabilities, often for neural network classification. Python example: calibrate a linear SVM Cross validation ensures that calibration uses predictions from folds not used to fit that fold's base model . Scaling is inside the pipeline so preprocessing is also fit only on the relevant training fold. What should you watch out for? Calibration is not classification accuracy: a well ranked classifier can be poorly calibrated and vice versa. Calibrating does not always improve test performance. Validate the change on unseen data and consider sampling uncertainty. Calibration and thresholds are different: calibrated probabilities can support cost based decisions, but you still choose a threshold separately. Changing ranking is possible: strictly monotonic sigmoid mappings preserve within model rank order, while isotonic mappings can create tied scores. In short: a calibrated classifier provides probabilities that can be interpreted as estimated event frequencies , enabling more reliable decisions when actual risk matters. References scikit learn: Probability calibration scikit learn: CalibratedClassifierCV

How do you choose a classification threshold?

Choose a classification threshold based on the cost of wrong decisions and the performance you need on unseen data—not automatically at 0.5. The threshold converts a model's probability or score into a final positive/negative decision. What does the threshold do? For a probability based binary classifier: A threshold of 0.5 is a common starting point, but not universally optimal. Lowering it tends to catch more positive cases while creating more alerts; raising it generally reduces alerts but can miss positives. Worked example: deciding which fraud alerts to investigate Suppose 100 transactions are scored by a fraud model. Consider two candidate thresholds: Threshold 0.70: TP=20, FP=5, FN=10, TN=65. Threshold 0.40: TP=27, FP=15, FN=3, TN=55. Assume each missed fraud costs ₹1,000 , while each false alert costs ₹100 in investigation effort. Threshold 0.40 is preferable under these assumed costs , despite producing more false alerts. If the relative costs changed, the preferred threshold might change too. These examples are hypothetical—not universal business cost estimates. How should you select a threshold? 1. Define your positive class and operational goal. Is a missed fraud worse than a false alarm? Are investigators limited to a certain number of cases per day? 2. Pick a metric or cost rule. Examples include minimizing expected misclassification cost, maximizing F1, or achieving at least 95% recall while keeping alert volume manageable. 3. Collect validation predictions. Use out of sample probabilities or decision scores; avoid selecting a threshold on the same examples used to fit the classifier. 4. Evaluate several candidate thresholds. For each, calculate TP, FP, FN, TN and the relevant metric or total cost. 5. Choose the threshold using validation data only , then evaluate the selected model and threshold once on a separate, untouched test set. 6. Monitor after deployment. Class prevalence, model calibration, and error costs may shift over time. Python example: tune the threshold with cross validation TunedThresholdClassifierCV searches thresholds using cross validation. Its default 5 fold approach avoids directly optimizing a threshold on the very same examples used to train each fold's underlying classifier. The final test set remains separate. Important cautions A better threshold does not improve underlying ranking quality: changing only the cutoff does not change ROC AUC for unchanged scores. Probabilities and decision scores differ: 0.5 is a probability cutoff, while some models' decision functions use 0 as the standard cutoff. Calibration matters for probability based cost rules: poorly calibrated probabilities may undermine theoretically justified cost thresholds. Avoid tuning against test labels or reusing training predictions: either leaks information and can overstate performance. In short: choose a threshold that matches the cost of false positives and false negatives or a clear performance constraint, tune it on validation data, and verify it on untouched test data. References scikit learn: Tuning the decision threshold scikit learn: Cost sensitive decision threshold example

What is macro averaging in multiclass evaluation?

Macro averaging evaluates a multiclass classifier by calculating a metric separately for each class and then taking the ordinary (unweighted) mean of those class scores. It gives every class equal influence , regardless of whether a class contains thousands of examples or only a few. How does it work? For a metric such as precision, recall, or F1 score: 1. Treat each class in turn as the positive class and all other classes as negative (one versus rest). 2. Compute the metric for each class. 3. Add the per class results and divide by the number of classes. Here K is the number of evaluated classes. Macro F1 means averaging the individual class F1 scores ; it is not generally the F1 calculated from macro precision and macro recall. Beginner friendly numerical example Imagine an eight photo classifier with three categories: Cat , Dog , and Bird . Actual: Cat, Cat, Cat, Cat, Dog, Dog, Bird, Bird Predicted: Cat, Cat, Cat, Cat, Cat, Dog, Dog, Bird Treat each class as positive in turn: Cat: TP=4, FP=1, FN=0. Precision=4/5=0.80; recall=4/4=1.00; F1=8/9≈0.889 . Dog: TP=1, FP=1, FN=1. Precision=1/2=0.50; recall=1/2=0.50; F1=0.500 . Bird: TP=1, FP=0, FN=1. Precision=1/1=1.00; recall=1/2=0.50; F1=2/3≈0.667 . Now average the three F1 scores: All three classes contribute one third to the final macro score. Even though the model correctly classifies all four cats, it cannot hide weaker performance on dogs or birds. Python example When is macro averaging useful? Imbalanced classes: when a rare category matters as much as a common one. Fair class level assessment: when you want to see whether the classifier performs reasonably across every label. Model comparison: when business requirements value per class performance rather than only total correct predictions. Common pitfalls It does not account for class frequency. Poor results on a very rare class can heavily affect the overall score, even if that class is not operationally important. Always inspect per class scores too. A single macro number can still conceal one badly performing class. Undefined per class metrics require a policy. In scikit learn, zero division controls cases such as no positive predictions. Do not confuse macro with weighted or micro averaging. Those use class support or pooled counts and are covered in the next curriculum question. In short: macro averaging gives one equal vote to each class , making it especially useful when minority class performance should not be drowned out by majority classes. References scikit learn: Multiclass and multilabel evaluation scikit learn: precision recall fscore support

What is micro averaging in multiclass evaluation?

What is micro averaging in multiclass evaluation? Micro averaging combines the true positives (TP), false positives (FP), and false negatives (FN) across all classes before computing precision, recall, or F1 score. Every individual prediction contributes to the overall count, so frequent classes usually have more influence. How it works 1. Evaluate each class using a one vs rest view. 2. Add the TP counts across classes; do the same for FP and FN. 3. Calculate: Micro precision = total TP / (total TP + total FP) Micro recall = total TP / (total TP + total FN) Micro F1 = 2 × micro precision × micro recall / (micro precision + micro recall), when the denominator is nonzero. Beginner friendly example Suppose a model classifies 10 emails into Work, Personal, or Promotions . It correctly predicts 8 emails and misclassifies 2. For single label multiclass classification, counting all three classes gives: Total TP = 8 Total FP = 2 Total FN = 2 Micro precision = 8 / (8 + 2) = 0.80 Micro recall = 8 / (8 + 2) = 0.80 Micro F1 = 0.80 Important: When every class is included in a single label multiclass task, micro precision, micro recall, micro F1, and accuracy are equal. That equality does not generally hold for multilabel classification or when reporting only a subset of labels. When is it useful? When you want one metric summarizing performance across all predictions. When errors on individual examples carry roughly equal importance. For multilabel problems, to pool decisions across labels. Limitation A model that performs very well on a large class but poorly on a rare class can still have a strong micro score. Inspect per class results and macro scores alongside it when minority class performance matters. Python example Key takeaway: Macro averaging gives each class equal weight; micro averaging pools the decisions and is driven by total counts. Reference: scikit learn: precision recall fscore support

What is weighted averaging in multiclass evaluation?

What is weighted averaging in multiclass evaluation? Weighted averaging summarizes a multiclass precision, recall, or F1 score by giving each class a weight proportional to its support —the number of true examples belonging to that class. How is it calculated? 1. Find each class's precision, recall, or F1 score. 2. Count its support (actual examples of that class). 3. Multiply its metric by its support, add across classes, and divide by the total support. Weighted metric = Σ(class metric × class support) / Σ(class support). Beginner friendly numerical example A classifier predicts the labels Cat , Dog , and Bird for 100 photographs: Cat: support 70 , F1 = 0.90 Dog: support 20 , F1 = 0.70 Bird: support 10 , F1 = 0.40 Weighted F1 = (70 × 0.90 + 20 × 0.70 + 10 × 0.40) / 100 = (63 + 14 + 4) / 100 = 0.81 . This illustrates the weighting arithmetic with illustrative per class scores. In practice, compute the scores from predictions, then weight them by observed class support. Why use it? It reflects the dataset's actual class distribution. It provides one summary when common categories contribute more examples. It can be useful for monitoring overall model performance on imbalanced datasets. Important limitations A high weighted score can hide weak performance on a rare but important class —here Bird's F1 is only 0.40. Weighted recall equals accuracy in ordinary single label multiclass classification when all labels are included; weighted precision and weighted F1 do not generally equal accuracy. Weighted F1 is not necessarily the harmonic mean of weighted precision and weighted recall , because each class's F1 is computed first and then averaged. If classes have different business costs , class support alone does not capture those costs. Check per class results and consider cost sensitive evaluation. Python example With average="weighted", the returned support value is None; per class support is available with average=None. Remember: Weighted averaging answers, “How well does the model perform when classes contribute in proportion to how frequently they appear in the test set?” Reference: scikit learn precision recall fscore support.

What is a confusion matrix in multiclass classification?

A multiclass confusion matrix shows which classes a model predicts correctly—and exactly which other classes it mistakes them for. For three classes, it has three rows and three columns. How to read it Rows = actual (true) classes ; columns = predicted classes in scikit learn's default convention. Some other displays reverse the axes, so always read their labels. Diagonal cells (same actual and predicted class) count correct classifications. Off diagonal cells count misclassifications. The cell at row Dog , column Cat means a real dog was predicted as a cat. The sum of a row tells you the number of actual examples of that class (its support ). Beginner friendly numerical example A model classifies 12 photos as Cat, Dog, or Bird. Instead of a table, read each row as a list: Actually Cat (5 photos): predicted Cat 4 , Dog 1 , Bird 0 . Actually Dog (4 photos): predicted Cat 1 , Dog 2 , Bird 1 . Actually Bird (3 photos): predicted Cat 0 , Dog 1 , Bird 2 . The diagonal gives 4 + 2 + 2 = 8 correct predictions. The other 4 are mistakes, so accuracy = 8 / 12 = 66.7% . What can you diagnose? One cat was mistaken for a dog. One dog was mistaken for a cat and another for a bird. One bird was mistaken for a dog. Dogs have 2 correct out of 4 actual dogs , suggesting this class deserves closer inspection. Python example When does it help? Error analysis: find which class pairs are consistently confused. Imbalanced classes: compare each class separately rather than trusting overall accuracy alone. Model improvements: investigate ambiguous training images, overlapping features, or labeling mistakes. Watch out: large raw counts may reflect larger class sizes. Use confusion matrix(y true, y pred, labels=labels, normalize='true') to normalize each row by that class's support and compare class specific error rates. A zero support row requires care when interpreting normalized results. Key takeaway: A multiclass confusion matrix tells you not just how many predictions are wrong, but which actual classes become which incorrect predictions . Reference: scikit learn confusion matrix.