Machine Learning

Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.

Why is logistic regression used for classification despite its name?

Logistic regression is called “regression” because it models how the log odds of an outcome change with input features—even though we usually use its predictions to classify observations. It belongs to the family of generalized linear models (GLMs), not ordinary least squares regression. What is being modeled? Suppose p is the probability that a customer cancels a subscription. A binary logistic regression fits: p / (1 p) is the odds of churn. log(p / (1 p)) is the log odds , also called the logit . The model expresses these continuous log odds as a linear function of the features . That relationship is the “regression” part of the name. The term logistic refers to the logistic (sigmoid) function used to convert the linear score back into a probability between 0 and 1. How does it become classification? A classifier turns the predicted probability into a label using a decision rule, often a threshold of 0.5 for binary classification. That threshold is a choice, not a property of regression itself. Practical example Imagine a fitted churn model that uses the number of missed payments (x): With x = 1, log odds = 1, so the estimated churn probability is about 0.27 → classify as Stay at a 0.5 threshold. With x = 3, log odds = 1, so the estimated churn probability is about 0.73 → classify as Churn . The model is not regressing directly on class names or fitting a straight line through binary labels . It fits a relationship on the log odds scale, estimates probabilities, and then makes classification decisions. Key distinction Linear regression: directly models a numerical outcome, often with squared error fitting. Logistic regression: models a categorical outcome's probability through a logit link, commonly fit with a likelihood based or log loss objective. Logistic regression can also be extended to multiple classes using multinomial models. In short: the mathematics is a regression model for log odds; the application is classification using predicted probabilities. Further reading scikit learn: Logistic regression and generalized linear models Penn State STAT 504: Binary logistic regression

What is the logistic or sigmoid function in logistic regression?

The logistic (sigmoid) function converts any real valued score into a number strictly between 0 and 1. In binary logistic regression, that number is interpreted as the model's estimated probability of the positive class. Formula Here, z is the linear score computed from the input features and learned coefficients. exp is the exponential function. How to read its S shaped curve When z = 0, sigmoid(z) = 0.50. When z is positive , the output is above 0.50 and approaches 1 as z increases. When z is negative , the output is below 0.50 and approaches 0 as z decreases. The curve is smooth and always increasing; it never actually reaches 0 or 1 for a finite z. Worked example Suppose a fitted logistic regression model computes a customer churn score z: A score of 2 therefore corresponds to an estimated churn probability of about 88.1% , under the fitted model . A subsequent decision threshold converts that probability into a class label. Why use the sigmoid? A raw linear score might be 4 or +7, neither of which is a valid probability. Sigmoid smoothly maps these scores to the probability scale while preserving their ordering. Important distinction: sigmoid gives an output in the probability range; it does not guarantee that the model's probabilities are well calibrated on unseen data. Calibration must be assessed separately. The sigmoid is also called the inverse logit . Logistic regression's log odds relationship will be explained in the next question. In short: logistic regression uses the sigmoid to translate an unbounded linear score into a bounded, interpretable class probability estimate. References Google Machine Learning Crash Course: The sigmoid function SciPy: scipy.special.expit

What are odds and log-odds in logistic regression?

Odds compare the chance that an event happens with the chance that it does not happen. Log odds are the natural logarithm of those odds. Binary logistic regression models log odds as a linear function of its input features. 1. Probability versus odds If the probability of a customer churning is p, then: For example, if p = 0.80: Probability of churn = 80% . Probability of not churning = 20% . Odds of churn = 0.80 / 0.20 = 4, or 4 to 1 . An 80% probability is therefore not the same as 80% odds . Odds are ratios, not percentages. 2. What are log odds? Taking the natural logarithm of the odds gives the logit : At p = 0.50, odds = 1, and log odds = 0. At p = 0.80, odds = 4, and log odds = ln(4) ≈ 1.386. At p = 0.20, odds = 0.25, and log odds = ln(0.25) ≈ 1.386. While probability stays between 0 and 1, log odds can take any real value. This lets logistic regression model them using a linear expression: 3. Why does this distinction matter? Imagine a model starts with a churn probability of 20% : Doubling the odds increased the probability from 20% to about 33.3%—not to 40%. For a one unit increase in a predictor, a logistic coefficient of ln(2) ≈ 0.693 means the odds multiply by 2 , assuming the other features are fixed. It does not imply that the probability doubles. In short: probability tells us how likely an event is; odds compare it with the alternative; log odds provide the unbounded scale on which logistic regression is linear. References UCLA Statistical Consulting: Interpreting odds ratios in logistic regression Penn State STAT 504: Binary logistic regression

How are logistic regression probabilities converted into class predictions?

Logistic regression converts an estimated probability into a class label by applying a decision threshold. In binary classification, the model first estimates the probability of one class, then uses a rule to choose between the two labels. 1. Obtain the predicted probability After training, logistic regression computes a linear score from the features and passes it through the sigmoid function: For labels 0 = Stay and 1 = Churn, p represents the model's estimated probability of Churn . 2. Apply a decision threshold With a threshold of 0.50, the model assigns Churn when the probability is above 0.50; otherwise it assigns Stay. This is the default cutoff used by scikit learn's binary logistic classifier (at an exact tie, scikit learn selects the first class in classes ). Worked example Suppose a model predicts these churn probabilities: Customer A: p = 0.18 → Stay (0). Customer B: p = 0.62 → Churn (1). Customer C: p = 0.87 → Churn (1). The probabilities carry more information than the final 0/1 labels: customers B and C are both classified as churn risks, but C receives a higher score. Python example with scikit learn predict proba(X new) returns one probability per class; columns follow clf.classes . predict(X new) returns labels directly using the classifier's built in decision rule. If your labels are not 0 and 1, check clf.classes before selecting a probability column. Is 0.50 always best? No. A business may choose a different threshold based on validation results and the relative costs of missed churn versus unnecessary retention offers. The model's estimated probabilities do not change when you change only the threshold; the predicted labels change. In short: logistic regression predicts a probability; a decision threshold converts that probability to a class label. The threshold is a decision rule—not something the sigmoid itself forces. References scikit learn: Tuning the decision threshold for class prediction scikit learn: LogisticRegression predict and predict proba

What is a decision threshold in classification?

A decision threshold is the cutoff used to convert a binary classifier's predicted probability or decision score into a class label. The classifier estimates how likely an event is; the threshold determines when to label it positive . How does it work? Suppose a fraud detector estimates the probability p that a transaction is fraudulent. If we choose a threshold of 0.70, the rule is: Practical example For three transactions: Transaction A: fraud probability 0.12 → Legitimate. Transaction B: fraud probability 0.58 → Legitimate. Transaction C: fraud probability 0.91 → Fraud. At this chosen 0.70 cutoff, only C is flagged. With a 0.50 cutoff, B would also be flagged. The underlying probability estimates have not changed; only the final decisions have. Why isn't 0.50 always best? A 0.50 probability cutoff is a common default, but its suitability depends on the consequences of mistakes: Lower threshold: typically catches more actual positives ( higher recall ), but may flag more negatives incorrectly ( more false positives ). Higher threshold: typically flags fewer cases, often reducing false positives but missing more positives ( more false negatives ). For fraud detection, missed fraud and unnecessary transaction blocks carry different costs. Choose the rule based on those costs rather than assuming 0.50 is optimal. How should the threshold be selected? 1. Define the positive class and what false positives and false negatives cost. 2. Compare candidate thresholds using validation data with metrics such as precision, recall, F1 score, or an explicit business cost function. 3. Fix the chosen threshold before assessing results on a held out test set . Avoid selecting it using test labels. Important: Some classifiers output decision scores instead of probabilities; their thresholds may be on a different scale (such as 0 for a decision function). A probability between 0 and 1 does not automatically guarantee good calibration. In short: a decision threshold converts model scores into actions, letting you align classification behavior with practical costs and requirements. References scikit learn: Tuning the decision threshold for class prediction scikit learn: Precision recall curve

How does changing the classification threshold affect predictions?

Changing the classification threshold changes which observations receive a positive label. It does not change a trained model’s predicted probabilities or its ranking of observations; it changes the decisions made from those scores. What happens when the threshold moves? For a binary classifier that predicts Positive when p = threshold: Lower threshold: more examples are labeled positive. True positives and false positives can increase; recall cannot decrease on the same fixed dataset. Higher threshold: fewer examples are labeled positive. False positives and true positives can decrease; recall cannot increase on that same dataset. Precision can move in either direction as the threshold changes, because both the numerator (true positives) and denominator (all predicted positives) can change. F1 score can also rise or fall. Worked example: fraud detection Assume eight transactions have these fixed fraud scores and actual labels: There are four fraudulent and four legitimate transactions. Threshold = 0.70 Flag A, B, and C. True positives = 2; false positives = 1; false negatives = 2; true negatives = 3. Precision = 2/3 ≈ 66.7%; recall = 2/4 = 50%. Threshold = 0.50 Flag A, B, C, D, and E. True positives = 3; false positives = 2; false negatives = 1; true negatives = 2. Precision = 3/5 = 60%; recall = 3/4 = 75%. Threshold = 0.30 Flag A through G. True positives = 4; false positives = 3; false negatives = 0; true negatives = 1. Precision = 4/7 ≈ 57.1%; recall = 4/4 = 100%. In this example, lowering the threshold catches more fraud but also blocks more legitimate transactions. In other datasets, precision need not decline steadily as the threshold falls. What stays the same? Predicted scores and their ranking stay the same when only the threshold changes. A fixed model’s threshold independent ranking summaries, such as ROC AUC and average precision , do not change just because you select a different operating cutoff. The confusion matrix, precision, recall, F1 score, and predicted class labels can change. How should the threshold be chosen? Evaluate candidate thresholds on a validation set , considering the actual costs of false positives versus false negatives. Fix the chosen threshold before final evaluation on untouched test data. Avoid tuning it on the test set. In short: changing the threshold trades off how many positives are caught against how many negatives are mistakenly flagged . Select the operating point that fits the application's objectives, not automatically 0.50. References scikit learn: Tuning the decision threshold for class prediction scikit learn: Precision recall curve

What is a decision boundary?

A decision boundary is the dividing surface in a classifier's feature space where its predicted class changes. One side is assigned one class, and the other side another. It need not perfectly separate the actual classes in the data. How can you picture it? With two input features , a decision boundary can be drawn as a line or curve on a 2D plot. With three features , it may be a plane or curved surface. With more features , it exists in a higher dimensional feature space that is harder to visualize directly. Worked example: logistic regression Suppose a trained churn classifier uses: x1: number of missed payments. x2: number of recent complaints. Its linear score is: With a 0.50 probability threshold , the model predicts Churn when z = 0. The decision boundary is where z = 0: Consider these examples: (2 missed payments, 0 complaints): z = 1; p ≈ 0.27 → predict Stay. (2 missed payments, 4 complaints): z = 1; p ≈ 0.73 → predict Churn. (2 missed payments, 2 complaints): z = 0; p = 0.50 → lies on the boundary . Which label wins a tie depends on the classifier's decision rule. The straight line x1 + 0.5 x2 = 3 separates the model's two prediction regions. This line is not proof that the true customer outcomes are perfectly separable. Does every classifier have a straight boundary? No. Linear logistic regression (on its supplied features) and linear SVMs have linear boundaries. Models such as decision trees, K nearest neighbors, and kernel SVMs can produce more complex, nonlinear boundaries. Decision boundary vs. decision threshold Decision threshold: a cutoff applied to the model's probability or score, such as 0.50. Decision boundary: the locations in feature space where the predicted label changes. Changing the threshold can move the boundary even when model coefficients are unchanged. For example, at threshold 0.80, this logistic model's boundary becomes z = ln(0.8/0.2) ≈ 1.386, rather than z = 0. In short: the decision boundary shows where predictions switch classes in feature space; the threshold specifies the score cutoff that determines that boundary. References scikit learn: DecisionBoundaryDisplay scikit learn: Classification boundaries with different SVM kernels

What is k-Nearest Neighbors (KNN)?

k Nearest Neighbors (KNN) is a supervised, instance based machine learning algorithm that predicts an outcome by looking at the K most similar training examples to a new observation. Unlike logistic regression, KNN does not fit a single equation for the decision boundary. It retains the labeled training examples (or an index of them) and uses them when making predictions. How does KNN classification work? 1. Choose K: the number of nearby examples to consider (such as K = 5). 2. Measure distance: compare the new observation with training examples using a distance measure, commonly Euclidean distance. 3. Find neighbors: identify the K closest examples. 4. Vote: predict the class occurring most often among these neighbors. Some variants give closer neighbors greater weight. Worked example: identifying a flower Suppose a flower classifier uses measurements such as petal length and petal width . For a new flower, its 5 nearest labeled flowers are: Setosa, Setosa, Versicolor, Setosa, Versicolor. With K = 5 and equal voting weights: Setosa votes: 3 . Versicolor votes: 2 . Prediction: Setosa . Because neighboring flowers can belong to different classes, KNN can form nonlinear decision boundaries without explicitly fitting a nonlinear equation. Why does feature scaling matter? Distance depends on feature units. If one feature ranges from 0–100,000 and another from 0–1, the large scale feature can dominate an unscaled Euclidean distance calculation. Scale distance based inputs using statistics learned from the training set , then apply the same transformation to validation and test data. Strengths and limitations Strength: intuitive, little conventional model fitting, and can adapt to irregular class shapes. Trade off: prediction may require costly neighbor searches and storage of training examples. Sensitivity: noisy points, irrelevant features, poor scaling, and very high dimensional data can make distance comparisons less useful. K matters: small K can be sensitive to noise; very large K may oversmooth local patterns. Choose K using validation data. KNN is not K means: KNN uses labeled examples for supervised predictions; K means groups unlabeled observations into clusters. In short: KNN predicts from nearby known examples— the neighborhood, the distance measure, and K determine its behavior. Reference scikit learn: Nearest Neighbors user guide scikit learn: KNeighborsClassifier API

How does KNN classification work?

KNN classification predicts a new observation’s class from the labels of its K nearest training examples. The key operation happens at prediction time: measure distances, select neighbors, and combine their votes. Step by step workflow 1. Prepare labeled data. Each training record has numerical feature values and a known class. 2. Choose K and a distance metric. For example, use K = 3 and Euclidean distance. 3. Measure distances from a new point to the stored training points. 4. Select the K nearest points and predict the class with the most votes. For two dimensional features, Euclidean distance is: where (x1, x2) is the new point and (a1, a2) is a training point. Worked example: classify a new customer Assume two customer features are measured on comparable scales . A new customer has features (2, 2). The labeled training examples are: (2, 3) → Renew ; distance = 1.000. (3, 2) → Renew ; distance = 1.000. (3, 3) → Leave ; distance = sqrt(2) ≈ 1.414. (0, 0) → Leave ; distance = sqrt(8) ≈ 2.828. (5, 5) → Leave ; distance = sqrt(18) ≈ 4.243. With K = 3, the closest labels are Renew, Renew, Leave : Renew receives 2 votes . Leave receives 1 vote . Predicted class: Renew. An unweighted neighbor vote fraction for Renew is 2/3 ≈ 66.7%; this is not automatically a calibrated probability of renewal. Try it in Python Practical details that matter Scaling: when features use different units, fit a scaler on training data and apply it consistently; otherwise one feature may dominate distance. Distance weighting: weights='distance' makes nearer examples count more than farther ones; it may change the winning class. Ties: class vote ties or equal distance neighbors near the Kth position need a well defined tie handling strategy. Prediction cost: KNN must search stored examples or a neighbor index when a query arrives; it does not simply evaluate a fixed linear formula. In short: KNN makes a local prediction by computing distances and voting among neighbors. The choice of K , distance metric, weighting, and preprocessing controls the result. References scikit learn: Nearest Neighbors—Classification scikit learn: KNeighborsClassifier

How do you choose K in KNN?

Choose K in k Nearest Neighbors (KNN) using validation performance, not a fixed formula. K is the number of neighbors that vote on a prediction; the best value depends on the data, distance measure, and the errors you care about. Why does K matter? Small K (for example, 1): predictions respond to very local patterns, but one mislabeled or noisy point can change the answer. This tends toward higher variance . Large K: votes average over a wider neighborhood, smoothing noise but potentially overlooking local class structure. This tends toward higher bias . K must not exceed the number of available training examples for a prediction (including the smallest training fold during cross validation). Practical way to select K 1. Reserve a test set for the final unbiased check; do not use it to tune K. 2. Scale features inside each training fold , for example with a StandardScaler in a pipeline. Otherwise large unit features may dominate distance. 3. Try multiple values (such as 1, 3, 5, 7, 9, 11, 15) using stratified cross validation for ordinary classification data. 4. Choose a relevant metric: accuracy for reasonably balanced classes, or F1 / balanced accuracy when error types and class imbalance matter. 5. Select based on cross validated performance , inspect variability across folds, and finally evaluate once on the untouched test set. Python example (scikit learn) The exact best K is computed from the data ; it is not assumed to be 3, 5, or another universal default. Common interview traps “Always use an odd K” is not a rule. Odd values reduce simple two class vote ties under uniform voting, but do not guarantee tie free outcomes in multiclass settings. sqrt(number of samples) is only a heuristic , not a replacement for validation. K in KNN is different from K in K means: KNN chooses how many labeled neighbors vote ; K means chooses how many clusters to form. In short: pick K by cross validation with leakage safe preprocessing and a task appropriate metric. Prefer a stable value that generalizes well—not the K that merely scores best on training data. References scikit learn: Nearest Neighbors scikit learn: GridSearchCV scikit learn: Pipelines