Machine Learning
Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.
How can heteroscedasticity affect regression analysis?
Heteroscedasticity can make regression uncertainty estimates unreliable because the spread of the errors changes across observations. It does not, by itself, mean that ordinary least squares (OLS) coefficients are biased. 1. Standard errors can be misleading Conventional OLS standard errors assume constant error variance. When errors have unequal variance, those standard errors may be too large or too small. 2. Confidence intervals and hypothesis tests may be wrong Because classical t tests, p values, and confidence intervals depend on standard errors, heteroscedasticity can lead to misleading conclusions about which predictors are statistically significant. 3. OLS may be less efficient If the usual exogeneity conditions hold, OLS coefficients can remain unbiased and consistent even with heteroscedasticity. However, they are no longer guaranteed to be the most efficient linear unbiased estimates. 4. Prediction uncertainty varies Suppose a house price model predicts low priced homes fairly closely but makes much larger errors on expensive homes. One fixed error band could understate uncertainty for expensive properties and overstate it for cheaper ones. How can you address it? Diagnose the pattern: inspect residuals versus fitted values or relevant features. Use heteroscedasticity robust standard errors: improve coefficient inference without necessarily changing fitted predictions. Consider weighted least squares: give observations different weights when there is a defensible model of error variance. Revisit the model or target transformation: a missing relationship or inappropriate functional form may explain the changing spread. In short, heteroscedasticity mainly threatens the reliability of conventional standard errors, significance tests, and uncertainty intervals. OLS coefficients are not automatically biased; the appropriate remedy depends on whether the goal is inference or prediction.
How do you compare two regression models fairly?
To compare two regression models fairly, evaluate both on the same unseen observations, with the same target and the same metrics. Otherwise, a better score may reflect an easier dataset rather than a better model. 1. Use identical evaluation data Train and tune the models without using the final test set. Evaluate both on the same test examples , in the same target units. For time based predictions, use a chronological split; for grouped records, keep related observations from leaking across splits. 2. Keep preprocessing leakage free Fit imputation, scaling, feature selection, and other learned preprocessing steps using training data only. Apply the fitted transformations to validation and test data. If models need different preprocessing, give each model its own pipeline while preserving the same evaluation split. 3. Choose a primary metric before comparing Select the metric based on the real world impact of prediction errors: MAE: average absolute error in target units. RMSE: emphasizes occasional large mistakes. R²: provides an additional view of fit relative to a mean baseline. Use the same metric definitions and calculation method for both models. 4. Compare against a simple baseline A model should improve meaningfully over a basic prediction rule, such as always predicting the training set mean or median. A more complicated model is not necessarily worthwhile if its gain is tiny. 5. Look beyond one overall score Check errors for important subgroups, unusually large mistakes, and residual patterns. Also consider inference time, model size, maintenance cost, and interpretability. Practical example Suppose two house price models are tested on the same 200 homes : Model A: MAE = ₹4.0 lakh . Model B: MAE = ₹3.7 lakh . Model B has lower average absolute error on that test set. But before choosing it, check whether the improvement is consistent across validation folds, whether it makes expensive outlier mistakes, and whether its extra complexity is justified. Avoid selecting the winner by repeatedly checking the test set Use validation or cross validation for model selection and hyperparameter tuning. Reserve the final test set for an unbiased comparison after choices are finalized. In short, a fair regression comparison keeps the evaluation data and metrics consistent, prevents leakage, checks performance against a baseline, and considers whether the improvement matters in practice.
What is a baseline regression model?
A baseline regression model is a simple prediction method used as a reference for evaluating a more sophisticated regression model. It answers an important question: Does the trained model predict better than a basic approach? Common regression baselines Mean baseline: predict the average target value from the training set for every new observation. This is a natural reference for squared error metrics such as MSE or RMSE. Median baseline: predict the training set median for every observation. This is a natural reference for MAE. Simple rule based baseline: use an obvious domain rule, such as the most recent observed value when forecasting a time series. Simple example Suppose the training set average monthly sales is 150 units. A mean baseline model predicts: for every new case, regardless of its input features. If a trained regression model consistently achieves lower errors than this baseline on the same unseen data, it is providing useful predictive value beyond the simple reference. How to compare fairly Calculate the baseline statistic (such as the mean) using training data only . Generate predictions from the baseline and the trained model on the same validation or test observations . Compare them using the same evaluation metric . Check whether the improvement is meaningful for the actual business problem. Why is a baseline important? A complex model can look impressive in isolation while adding little value. If it cannot outperform a basic baseline, investigate the data, features, training setup, or whether the problem is predictable. A baseline is a minimum reference , not necessarily the performance level required for deployment. In short, a baseline regression model gives you a simple benchmark—often the training target mean or median—so you can tell whether a more complex model genuinely improves predictions.
What is classification in machine learning?
Classification in machine learning is a supervised learning task in which a model predicts which category (class) an observation belongs to. Unlike regression, which predicts numerical quantities, classification predicts discrete labels such as spam or not spam. Examples Email filtering: Spam or Not Spam. Loan risk: Default or No Default. Document sorting: Finance, Sports, or Technology. How does classification work? 1. Training: The model learns patterns from examples containing input features and their known class labels. 2. Prediction: For a new observation, it uses those patterns to predict a class. Many classifiers can also output a probability or score for each class. Example: A spam classifier learns from previously labeled emails. For a new email, it may estimate a 90% probability of spam and classify it as Spam using a chosen decision threshold. Main forms of classification Binary classification: choose between two classes, such as Yes and No. Multiclass classification: choose one class from three or more possibilities. Multilabel classification: assign multiple applicable labels to the same observation. Classification vs regression Classification: predicts a category, even if that category is encoded as a number like 0 or 1. Regression: predicts a numerical quantity, such as price, temperature, or revenue. How is a classifier evaluated? Common metrics include accuracy, precision, recall, and F1 score . The appropriate metric depends on which classification mistakes matter most. In short, classification teaches a model to assign observations to predefined categories using patterns learned from labeled training data.
When should a classification model be used?
Use a classification model when the outcome you want to predict is a category or class rather than a continuous numerical value. When is classification appropriate? The target is categorical: for example, Fraud / Not Fraud, Will Churn / Will Stay, or Low / Medium / High Risk. You have labeled examples: historical records include the correct class, allowing a supervised model to learn patterns. Input features contain useful signals: customer activity, transaction details, or message text help distinguish classes. The prediction supports a decision: for example, flag a suspicious payment or route a support ticket to the right team. Practical example Suppose a telecom company wants to identify customers likely to cancel their subscriptions. Input features: recent usage, complaints, plan type, and payment history. Target: Churn or No Churn. Output: the predicted class, often accompanied by an estimated probability of churn. A probability threshold can be selected based on the cost of missing a customer who will churn versus unnecessarily contacting one who will stay. When should you use something else? Regression: when the target is a quantity, such as monthly revenue or the number of units sold. Rule based logic: when the categories can be determined reliably by simple, explicit rules without learning from examples. Important: A class encoded as 0 or 1 is still categorical; numerical encoding alone does not make the task regression. What should you check before choosing a classifier? Ensure that labels are trustworthy, examples represent the cases the model will encounter, and the evaluation metric reflects the cost of different mistakes. Accuracy alone may be misleading for rare outcomes such as fraud. In short, choose classification when you need to assign observations to predefined classes using patterns learned from labeled data.
What is binary classification?
Binary classification is a supervised machine learning task in which a model predicts one of two possible classes for an observation. Typical class pairs include: Spam or Not Spam for an email. Fraud or Legitimate for a transaction. Churn or No Churn for a customer. How does it work? The model learns patterns from training examples that each have a known label, commonly encoded as 0 or 1. For a new observation, it predicts a class. Many binary classifiers also produce a score or estimated probability for the positive class. Example: fraud detection Suppose a classifier estimates that a transaction has a 0.82 probability of being fraudulent. With a decision threshold of 0.50: Probability at or above 0.50 → Fraud . Probability below 0.50 → Legitimate . The threshold is a decision rule, not a fixed requirement . A lower threshold may catch more fraud but flag more legitimate transactions. What are positive and negative classes? Positive class: the event you are trying to detect, such as Fraud. Negative class: the alternative, such as Legitimate. These labels are conventions; positive does not mean desirable. How is binary classification evaluated? Important outcomes include: True positive: fraud correctly flagged. False positive: legitimate transaction incorrectly flagged. False negative: fraud incorrectly missed. True negative: legitimate transaction correctly accepted. Accuracy alone can be misleading when one class is rare. Metrics such as precision, recall, F1 score, and ROC AUC provide additional views of performance. In short, binary classification assigns an observation to one of two classes. Choosing the prediction threshold and evaluation metrics depends on the cost of each type of mistake.
What is multiclass classification?
Multiclass classification is a supervised machine learning task in which a model assigns an observation to exactly one of three or more mutually exclusive classes . Simple example A customer support system may classify each new ticket as one of: Billing Technical Support Account Access Each ticket receives one predicted category, even though the model considers all three possibilities. How does it work? Training: learn patterns from examples labeled with one of the available classes. Prediction: compare the model's scores or estimated probabilities for the classes and select a class, often the one with the highest value. For example, a model might output: Billing: 0.15 Technical Support: 0.75 Account Access: 0.10 The predicted class is Technical Support . These numbers illustrate an example of a model that outputs class probabilities; not every classifier provides calibrated probabilities. How is it different from other classification tasks? Binary classification: exactly two possible classes, such as Spam and Not Spam. Multiclass classification: three or more possible classes, but only one is assigned per observation. Multilabel classification: an observation can receive several labels at once . How is it evaluated? Common measures include accuracy , a confusion matrix , and precision, recall, or F1 score computed per class . Macro averaged scores help reveal poor performance on smaller classes that overall accuracy may hide. In short, multiclass classification predicts one category from three or more mutually exclusive options, such as routing a support ticket to exactly one department.
What is multilabel classification?
Multilabel classification is a supervised machine learning task in which one observation can belong to multiple classes at the same time . Unlike multiclass classification, which selects exactly one category from several options, multilabel classification predicts which labels apply to each observation. It may assign none, one, or several labels. Simple example Suppose a customer support ticket says: “My delivery was late, and I want a refund.” The system could assign both: Delivery Issue Refund Request The labels are not mutually exclusive, so choosing only one would lose useful information. How does multilabel classification work? Training data: each example has a set of applicable labels rather than one exclusive class. Prediction: the model produces a score or estimated probability for each label. Decision: apply thresholds to decide which labels to assign. Thresholds may differ by label and should be tuned using validation data. For instance, a model might output: Delivery Issue: 0.92 → assign Refund Request: 0.81 → assign Technical Issue: 0.08 → do not assign These decisions assume a threshold of 0.50; the best threshold depends on the task and the costs of mistakes. Multiclass vs. multilabel Multiclass: a news article is categorized as exactly one of Sports, Business, or Politics. Multilabel: an article about sports sponsorship can receive both Sports and Business. How do you evaluate it? Useful metrics include precision, recall, and F1 score (micro or macro averaged). Subset accuracy is particularly strict: all predicted labels for an observation must exactly match its true label set. In short, use multilabel classification when categories can overlap and an observation may need several labels simultaneously.
What is logistic regression?
Logistic regression is a supervised machine learning algorithm commonly used for binary classification , such as predicting whether a customer will churn or a transaction is fraudulent. Instead of predicting an unrestricted numerical value, it estimates the probability that an observation belongs to a chosen class . Simple example Imagine a bank wants to predict whether a transaction is fraudulent. Inputs: transaction amount, location, transaction history, and time. Possible classes: Fraud and Not Fraud. Example output: probability of fraud = 0.82. If the bank uses a decision threshold of 0.50, it would classify this transaction as Fraud . The threshold can be adjusted according to the cost of false alarms and missed fraud. Core idea Logistic regression combines the input features into a linear score and passes that score through a logistic (sigmoid) function to produce a value between 0 and 1. Here, p is the predicted probability of the positive class, and z is the weighted combination of inputs. The formula and training process are explained in the next concept. Why is it useful? Simple and fast: often a strong baseline for classification. Interpretable: coefficients describe how features affect the model's log odds, subject to modeling assumptions. Probability output: supports threshold based decisions. Works well with suitable features: especially when the log odds are approximately linear in the inputs. Important limitation Basic logistic regression has a linear decision boundary in its feature space . It may miss complex nonlinear patterns unless features are transformed or interactions are added. In short, logistic regression is a classification method that converts a linear score into an estimated class probability, making it a practical starting point for binary classification problems.
How does logistic regression work?
Logistic regression learns a relationship between input features and the probability of a class. For binary classification, it fits weights so examples labeled 1 receive higher predicted probabilities and examples labeled 0 receive lower ones. Step 1: Calculate a linear score For each observation, combine the feature values with learned weights and an intercept: Each w is a feature coefficient; b is the intercept. The score z can be any real number. Step 2: Convert the score into a probability Pass z through the sigmoid function : This produces 0 < p < 1, interpreted by the model as the estimated probability of class 1. Step 3: Measure prediction error During training, compare the estimated probabilities with the known labels using binary cross entropy (log loss) : For an actual label of 1, predicting a very low p is penalized heavily. For an actual label of 0, predicting a very high p is penalized heavily. Step 4: Learn the weights An optimizer changes the coefficients and intercept to reduce the total training loss. Regularization, when used, discourages excessively large coefficients and can help generalization. The learning algorithm is not simply fitting a straight line to the 0/1 labels using squared error. Worked example: customer churn Suppose a simplified trained model uses missed payments (x) as its only feature: For x = 3, we get z = 0.4, so p = sigmoid(0.4) ≈ 0.60. Using a 0.50 decision threshold , the predicted label is Churn (1). In practice, choose the threshold on validation data based on the costs of false positives and false negatives. Key takeaway: logistic regression learns weights by minimizing log loss , transforms their linear score into a probability with the sigmoid, and then applies a decision rule to obtain a class label. Reference scikit learn: LogisticRegression documentation