Machine Learning
Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.
What is accuracy in classification?
Accuracy is the fraction of classification predictions that match the true labels. It answers: Out of all the examples, how many did the model classify correctly? Formula For a binary classifier: Here TP and TN are correct positive and negative predictions; FP and FN are incorrect predictions. More generally, for binary or multiclass classification: Multiply by 100 to express it as a percentage. Worked example: quality inspection A factory classifier labels products as Defective or Good . For 100 inspected products, its confusion matrix counts are: TP = 15: defective products correctly flagged. TN = 75: good products correctly cleared. FP = 5: good products incorrectly flagged. FN = 5: defective products missed. Interpretation: the classifier made correct predictions for 90 of 100 products . Accuracy includes both correctly detected defects and correctly identified good products. Python example The default normalize=True returns the fraction correct. With normalize=False, it returns the number of correct predictions. When is accuracy useful? Classes are reasonably balanced , and different error types have similar costs . You want an easy to explain overall correctness measure. You compare models using the same held out evaluation set . Important limitation Accuracy does not tell you which class the model gets wrong. For instance, if 99% of transactions are legitimate, a model that labels every transaction as legitimate gets 99% accuracy while detecting no fraud . This is why accuracy should be checked alongside the confusion matrix and other metrics, particularly for imbalanced or high stakes tasks. For multiclass classification, accuracy still counts the fraction of correct class labels. For multilabel classification, scikit learn's accuracy score instead uses exact match/subset accuracy : every label for an example must match. In short: accuracy measures total prediction correctness, not the ability to identify any one particular class or the cost of errors. References scikit learn: accuracy score API scikit learn: Classification model evaluation
When can accuracy be misleading?
Classification accuracy can be misleading when the classes are imbalanced, different mistakes have different consequences, or a model performs poorly on an important subgroup. A high percentage of correct predictions does not necessarily mean the classifier performs its intended job. 1. Class imbalance can hide failure Imagine a fraud detection model evaluating 1,000 transactions : 990 transactions are legitimate. 10 transactions are fraudulent. A model that predicts “legitimate” for every transaction produces: The model has 99% accuracy yet catches zero fraudulent transactions. Its fraud recall is TP / (TP + FN) = 0 / 10 = 0%. The same problem arises in rare disease detection, failure prediction, and other applications with a rare but important positive class. 2. The costs of errors may be unequal Accuracy treats false positives and false negatives as one mistake each , but their consequences may differ. In disease screening, missing a real case ( false negative ) may be much worse than requesting an unnecessary follow up ( false positive ). In spam filtering, wrongly hiding a legitimate business email ( false positive ) may be especially costly. A model with slightly lower accuracy can therefore be more useful if it makes fewer of the expensive mistakes. 3. Overall accuracy can conceal subgroup problems Suppose a classifier achieves 95% accuracy overall but is much less accurate for a small but important user segment. Looking only at the aggregate hides that weakness. Check performance by relevant populations, conditions, and time periods, using sufficiently large evaluation samples. What should you evaluate instead? Choose metrics according to the decision you need to make: Recall (sensitivity): How many actual positives did the model detect? Important when misses are costly. Precision: Of predictions labeled positive, how many were truly positive? Important when false alarms are costly. F1 score: Combines precision and recall in one number when both are relevant; does not account for true negatives. Balanced accuracy: Gives each class equal weight through its recall; for binary classification, it averages sensitivity and specificity. Confusion matrix: Shows the actual counts of TP, TN, FP, and FN instead of hiding them in one average. For the all legitimate fraud classifier, balanced accuracy is 50% , not 99%: sensitivity is 0% and specificity is 100%, so (0% + 100%) / 2 = 50%. Python example Practical rule Always compare a classifier with a simple baseline , inspect its confusion matrix, and choose metrics that reflect the rarity and business costs of errors. Evaluate on a representative held out dataset; improving the displayed accuracy alone is not enough. In short: accuracy answers “How often are predictions correct?”—not “Does the model find the cases that matter?” References Google Machine Learning Crash Course: accuracy, precision and recall scikit learn: balanced accuracy and classification evaluation
What is precision?
Precision is the proportion of predictions labeled positive that are actually positive. It answers: “When the classifier raises a positive alert, how often is it correct?” Formula TP (true positives): correctly predicted positive examples. FP (false positives): negative examples incorrectly predicted positive. TP + FP is the total number of positive predictions , not the total number of examples. Precision is also called positive predictive value (PPV) . Beginner friendly numerical example: fraud alerts A fraud model flags 50 bank transactions as suspicious. Investigators confirm that: 40 flagged transactions were actually fraudulent ( TP = 40 ). 10 flagged transactions were legitimate ( FP = 10 ). Interpretation: for every 10 fraud alerts, approximately 8 are genuine fraud and 2 are false alarms in this evaluated sample. What this does not tell you: how many fraudulent transactions the classifier missed . That requires recall, which uses false negatives and is covered in the next question. When does high precision matter? Precision is useful when false positive alerts are expensive , such as when positive predictions trigger human review, automatically block legitimate users, or interrupt an important workflow. High precision helps limit unnecessary alerts. However, a classifier can reach high precision by making very few positive predictions , so precision must be interpreted alongside the number of alerts and recall. Python example The examples outside the first 50 alerts do not change precision because they are predicted negative. Important details Positive class: decide which label means “positive”; with nonstandard labels, set pos label explicitly. Decision threshold: changing the threshold can change the number and composition of positive predictions, so precision can rise or fall. No positive predictions: if TP + FP = 0, precision is mathematically undefined. scikit learn exposes a zero division setting to handle that case. Multiclass tasks: define how classwise precision values are combined (for example, macro or weighted averaging) rather than assuming one binary class formula describes every class. In short: precision tells you how trustworthy positive predictions are , not how many of all actual positives were found. References Google ML Crash Course: Precision and recall scikit learn: precision score
What is recall?
Recall measures the fraction of actual positive cases that a classifier correctly identifies as positive. It answers: “Of all cases that truly matter, how many did the model find?” Formula TP (true positives): actual positives correctly detected. FN (false negatives): actual positives the model missed. TP + FN: the total number of actual positives, regardless of prediction. Recall is also called sensitivity or true positive rate (TPR) . For a fixed binary classification threshold, it ranges from 0 to 1 (0%–100%). Worked example: screening for defective parts A factory actually produces 50 defective parts . Its inspection model: Flags 45 defective parts correctly ( TP = 45 ). Misses 5 defective parts ( FN = 5 ). Interpretation: the model detects 9 out of every 10 defective parts in this sample. The other 10% slip through. The number of non defective parts incorrectly flagged ( false positives ) does not appear in the recall formula. Why is recall important? Recall matters especially when missing a real positive case is costly : Disease screening: missing a condition can delay follow up care. Fraud detection: missed fraud can cause financial loss. Equipment failure detection: missed warning signs may lead to downtime. High recall does not mean positive predictions are usually correct: a classifier can label nearly everything positive, catching all real positives while also generating many false alarms. Precision describes that other aspect. Python example Common pitfalls Recall is not precision: recall divides by actual positives (TP + FN), while precision divides by predicted positives (TP + FP). Threshold choice matters: reducing the threshold often raises recall but can create additional false positives; the precise trade off depends on the model and data. Define the positive class: when labels differ from 0/1, specify pos label. For multiclass evaluation, choose an appropriate classwise or averaging scheme. No actual positives: when TP + FN = 0, recall is undefined mathematically. In scikit learn, zero division controls its handling. In short: recall measures how successfully a classifier finds actual positives . If missed cases are especially dangerous or expensive, recall is a key metric—but should not be considered in isolation. References scikit learn: recall score Google Machine Learning Crash Course: recall
What is specificity?
Specificity measures the proportion of actual negative cases a binary classifier correctly predicts as negative. It answers: “Of the cases that truly do not belong to the positive class, how many did the model correctly clear?” It is also called the true negative rate (TNR) . Formula TN (true negatives): actual negatives correctly predicted negative. FP (false positives): actual negatives incorrectly predicted positive. TN + FP: the total number of actual negatives , not all evaluated cases. Worked example: quality control A manufacturing inspection system treats Defective as the positive class and Good as the negative class. Among 80 actually good products : 72 are correctly classified as good ( TN = 72 ). 8 are incorrectly flagged as defective ( FP = 8 ). Interpretation: the model correctly clears 9 out of every 10 good products in this sample. The remaining 10% are false alarms. Relationship to false positive rate The false positive rate (FPR) is the fraction of actual negatives incorrectly flagged as positive: Here, FPR = 8 / 80 = 10%, so specificity = 1 0.10 = 90%. Specificity versus recall Recall (sensitivity/TPR): Among actual positives, what fraction did we identify? TP / (TP + FN). Specificity (TNR): Among actual negatives, what fraction did we identify? TN / (TN + FP). These two metrics examine different actual classes. A model can achieve high specificity while missing many positives, so examine both when the distinction matters. Python example When is specificity useful? Specificity is important when you want to avoid false alarms among actual negatives , such as unnecessarily flagging good products or legitimate transactions. It is also one ingredient of balanced accuracy : for binary classification, balanced accuracy equals (recall + specificity) / 2. Caution: If there are no actual negatives (TN + FP = 0), specificity is undefined. Also, changing the classification threshold may change both specificity and recall. Define the positive class before calculating either metric. In short: specificity tells you how reliably a classifier recognizes actual negatives . Higher specificity means a lower false positive rate. References scikit learn: confusion matrix Google ML Crash Course: false positive rate
What is F1-score?
F1 score is the harmonic mean of precision and recall . It summarizes how well a classifier balances trustworthy positive predictions ( precision ) with finding actual positives ( recall ). Formula The F1 score ranges from 0 (worst) to 1 (best). Its harmonic mean penalizes an imbalance between precision and recall more than an arithmetic mean would. Beginner friendly numerical example A fraud detection system reviews 100 transactions and records: TP = 30: fraudulent transactions correctly flagged. FP = 10: legitimate transactions incorrectly flagged. FN = 20: fraudulent transactions missed. TN = 40: legitimate transactions correctly cleared. First calculate the component metrics: The arithmetic mean of precision and recall is 67.5% ; F1 is lower at 66.7% because the harmonic mean gives more influence to the smaller value. Why use the harmonic mean? Suppose precision = 1.0 but recall = 0.10 . The arithmetic mean is 0.55, which could sound reasonably good, but the F1 score is only: Even though every positive prediction is correct, the model misses 90% of actual positives. The lower F1 makes that weakness visible. Python example When is F1 score useful? Imbalanced classification: where overall accuracy can be dominated by the majority class. Information retrieval and fraud detection: when both missed positives and false alarms matter. Comparing models: when you want a single metric that gives equal mathematical weight to precision and recall. Limitations and common mistakes F1 does not include true negatives directly. If correctly recognizing negatives is important, inspect specificity and the confusion matrix too. Equal weighting may be inappropriate: F1 does not account for unequal real world costs of FP and FN; consider F beta or cost based evaluation where appropriate. Multiclass tasks require an averaging choice: macro F1 averages class scores equally, while weighted F1 weights them by class support. Undefined cases: when TP, FP, and FN are all zero, F1 is mathematically undefined; scikit learn's zero division argument controls the reported value. In short: F1 score combines precision and recall into one harmonic mean score, favoring models that do reasonably well on both rather than excelling at only one. References scikit learn: f1 score Google Machine Learning Crash Course: F1
How do precision and recall differ?
Precision and recall measure different aspects of positive class performance. Precision asks how reliable the model's positive predictions are; recall asks how many real positive cases it successfully finds. The key difference Precision: “Of everything predicted positive, how much is truly positive?” It is affected by false positives . Recall: “Of everything actually positive, how much did we detect?” It is affected by false negatives . Their formulas are: The numerators are the same ( true positives ), but the denominators are different : precision considers all predicted positives , whereas recall considers all actual positives . Worked example: fraud detection Imagine a fraud screening model evaluating 100 transactions : TP = 30: genuine fraud correctly flagged. FP = 10: legitimate transactions incorrectly flagged. FN = 20: genuine fraud the model missed. TN = 40: legitimate transactions correctly cleared. Interpretation: When the system raises an alert, 75% of alerts are genuine fraud; of all real fraud cases, it catches 60% . These are not two ways of describing the same result. When should you favor each metric? Favor precision when false alarms cost more. For example, if each fraud alert temporarily freezes a customer's account, you want flagged transactions to be trustworthy. Favor recall when missed cases cost more. For example, for early detection of dangerous equipment failures, finding as many genuine failures as possible may justify additional false alarms. A model may have high recall but low precision when it labels many examples positive. Another model may have high precision but low recall when it issues very few carefully selected alerts. Neither extreme is automatically best. Python example Common points of confusion Precision is not accuracy: accuracy counts correctly classified negatives too; precision focuses on predictions labeled positive. Recall is also called sensitivity or true positive rate. It is not specificity; specificity measures correctly recognized actual negatives. Metrics depend on the decision threshold: moving the threshold changes which examples count as positive. Precision and recall often trade off, but the change is data dependent; their trade off is explored in the next question. Define the positive class and the evaluation dataset before comparing scores. For multiclass problems, calculate per class metrics or specify an averaging method. In short: precision measures false alarm control among positive predictions; recall measures missed case control among actual positives. Choose based on which mistakes your application can tolerate. References Google ML Crash Course: Accuracy, precision and recall scikit learn: Classification metrics
What is the precision-recall trade-off?
The precision–recall trade off means that changing a binary classifier's decision threshold often improves one metric at the expense of the other. Lowering the threshold usually finds more real positives ( higher recall ) but also raises more false alarms, potentially reducing precision . Why does a threshold create a trade off? A classifier can output a score for how likely an example is positive. If its score is at least the threshold , predict the positive class. Higher threshold: fewer positive predictions. This usually reduces false alarms but can miss real positives. Lower threshold: more positive predictions. This catches more real positives but may also flag additional negatives. Important nuance: For a fixed set of scores, lowering the threshold cannot reduce recall (unless no actual positives exist, where recall is undefined). Precision is not guaranteed to decrease monotonically ; its behavior depends on which examples cross the threshold. Beginner friendly numerical example: fraud detection Imagine eight transactions , with four actual frauds and four legitimate transactions. Their model scores are: Fraud (positive): 0.95, 0.80, 0.55, 0.40. Legitimate (negative): 0.75, 0.45, 0.30, 0.10. A transaction is flagged when its score is greater than or equal to the threshold. At threshold 0.80: Flagged: fraud at 0.95 and 0.80; no legitimate transactions. TP=2, FP=0, FN=2. Precision: 2/(2+0)=100%. Recall: 2/(2+2)=50%. At threshold 0.40: Flagged: all four frauds, plus legitimate transactions at 0.75 and 0.45. TP=4, FP=2, FN=0. Precision: 4/(4+2)=66.7%. Recall: 4/(4+0)=100%. What changed? Lowering the threshold caught the two previously missed frauds, but also created two false alarms. Which threshold is preferable depends on the cost of missed fraud versus investigating legitimate transactions. Verify with Python How do you choose a threshold in practice? 1. Identify the business cost of false positives and false negatives. Missing a serious condition may be worse than an extra review; incorrectly blocking legitimate customers may be expensive. 2. Evaluate several thresholds on validation data with representative class frequencies. 3. Choose a threshold that meets a practical constraint, such as recall at least 95% , while keeping false alarms manageable. 4. Keep the test set untouched until final evaluation, and monitor performance after deployment if class prevalence changes. A precision–recall curve visualizes results across thresholds. It is covered separately later in the curriculum. Common misconception A lower threshold does not improve the model's underlying ranking of examples. It changes the decision rule applied to existing model scores. Also, precision may increase at some thresholds even as recall increases, so do not assume a strict inverse relationship at every step. In short: the precision–recall trade off is a practical decision about how many missed positives versus false alarms you can accept, usually controlled by the classification threshold. References Google Machine Learning Crash Course: Accuracy, precision, recall and trade offs Google Machine Learning Crash Course: Classification thresholds scikit learn: Precision recall curve
What is an ROC curve?
An ROC (Receiver Operating Characteristic) curve shows how a binary classifier's true positive rate (TPR) changes against its false positive rate (FPR) as the decision threshold changes. It evaluates a model's ability to distinguish classes across thresholds rather than at just one cutoff. What are the two axes? X axis: False positive rate (FPR) — among actually negative examples, the fraction incorrectly flagged positive. Y axis: True positive rate (TPR) — among actually positive examples, the fraction correctly detected. TPR is also recall or sensitivity . Each point on the ROC curve corresponds to a different score threshold. In general, lowering the threshold flags more items, increasing or keeping the same both TPR and FPR. Numerical example: detecting fraud Suppose we score six transactions , three fraudulent (positive) and three legitimate (negative): Fraud scores: 0.90, 0.70, 0.40 Legitimate scores: 0.80, 0.30, 0.10 Predict fraud when score = threshold: Threshold above 0.90: TP=0, FP=0 → (FPR=0, TPR=0) . Threshold 0.90: TP=1, FP=0 → (0, 1/3) . Threshold 0.80: TP=1, FP=1 → (1/3, 1/3) . Threshold 0.70: TP=2, FP=1 → (1/3, 2/3) . Threshold 0.40: TP=3, FP=1 → (1/3, 1) . Threshold 0.30: TP=3, FP=2 → (2/3, 1) . Threshold 0.10: TP=3, FP=3 → (1, 1) . Plotting these points in order creates a stair shaped ROC curve. The example demonstrates how changing the cutoff changes both detection and false alarms. How do you interpret the curve? Upper left corner (0, 1): ideal—detect all positives without incorrectly flagging negatives. Diagonal from (0, 0) to (1, 1): chance level discrimination on average. Curve closer to the upper left: generally stronger separation between positive and negative scores. Threshold choice: depends on the real costs of false alarms versus missed positives; no single ROC point is best for every application. The area under the ROC curve (ROC AUC) summarizes the curve in one number. It is a separate concept covered in the next question. Python example Use continuous model scores or positive class probabilities , not only the final 0/1 predictions, to obtain meaningful thresholds. Common mistakes ROC does not plot precision: it plots TPR against FPR . Precision–recall curves are different. ROC is not inherently threshold free: it aggregates performance across thresholds, but deploying the classifier still requires a threshold or decision rule. Imbalanced datasets: ROC curves can look promising even when false positive counts are operationally unacceptable. Inspect absolute errors and consider a precision–recall curve when positives are rare. Multiclass models: standard binary ROC calculation requires an adapted approach (for example, one versus rest per class). In short: an ROC curve illustrates the trade off between catching positives and generating false positives across decision thresholds. References scikit learn: roc curve Google Machine Learning Crash Course: ROC and AUC
What is ROC-AUC?
ROC AUC (Area Under the Receiver Operating Characteristic Curve) summarizes how well a binary classifier ranks positive cases above negative cases across decision thresholds . A larger area generally indicates better class discrimination. What does ROC AUC measure? An ROC curve plots true positive rate (recall) against false positive rate while varying the classification threshold. ROC AUC is the area under that curve, typically between 0 and 1. 1.0: perfect ranking of positive cases above negative cases. 0.5: chance level ranking (no discrimination). Below 0.5: positive cases tend to receive lower scores than negative cases; this is worse than chance with the current score direction. For binary classification, ROC AUC also has a useful ranking interpretation: the probability that a randomly selected positive receives a higher model score than a randomly selected negative, with ties counting as half . Beginner friendly numerical example Suppose a fraud model assigns these risk scores: Fraud (positive): 0.90 and 0.60. Legitimate (negative): 0.70 and 0.20. Compare each positive against each negative—there are 2 × 2 = 4 pairs: 0.90 0.70 → correctly ranked. 0.90 0.20 → correctly ranked. 0.60 < 0.70 → incorrectly ranked. 0.60 0.20 → correctly ranked. Three of four pairs are correctly ordered: Interpretation: On these observations, a randomly selected fraudulent transaction receives a higher risk score than a randomly selected legitimate transaction 75% of the time (counting tied scores as half a win). Python example Use continuous scores such as positive class probabilities or decision function values. Passing only final binary predictions discards most ranking information. When is ROC AUC useful? Comparing models' discrimination before selecting a classification threshold. Evaluating ranking quality when several operating thresholds are plausible. Summarizing performance over the full range of false positive rates. Important limitations ROC AUC does not determine a deployment threshold. A model with good AUC can still perform poorly at the specific false positive rate your application can tolerate. ROC AUC does not assess probability calibration. A score of 0.9 need not mean a 90% real world probability. Rare positives can require different metrics. On highly imbalanced tasks, investigate precision, recall, alert volume, and precision–recall curves; ROC AUC can look strong even when positive predictions produce many false alarms. Both classes are required for binary ROC AUC. Multiclass variants exist but require a specified aggregation strategy, such as one vs rest or one vs one. In short: ROC AUC evaluates how well a classifier ranks positives above negatives across thresholds, not how accurate its final labels are at one threshold. References scikit learn: roc auc score Google Machine Learning Crash Course: ROC and AUC