Machine Learning

Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.

How does least squares estimate regression coefficients?

Least squares estimates regression coefficients by choosing the coefficient values that minimize the total squared difference between the actual target values and the model's predictions. Start with the regression equation For simple linear regression: where: b0 is the intercept b1 is the slope x is the input feature ŷ is the predicted target Calculate the residuals For each training observation: Substituting the regression equation: Square and sum the residuals Least squares defines the objective: or: The goal is to find the values of b0 and b1 that make this quantity as small as possible. How are the best coefficients found? Mathematically, the squared error function can be minimized by finding where its slope with respect to the coefficients becomes zero. For simple linear regression, this produces: and: where x̄ and ȳ are the mean values of the input and target. Intuition behind the slope The slope becomes larger when x and y tend to increase together. It becomes negative when larger values of x tend to be associated with smaller values of y. The denominator measures how much the input feature varies, while the numerator measures how the input and target vary together. With multiple features For multiple linear regression: the same principle applies: choose all coefficients together so that the total squared residual is minimized. This can be solved using linear algebra or numerical optimization methods. Why this gives the best fitting line Among all possible coefficient combinations, least squares selects the one with the smallest total squared training error. This does not mean every prediction is exact. Some residuals remain, but their squared total is as small as possible for that linear model. In short, least squares estimates regression coefficients by adjusting them until the sum of squared residuals reaches its minimum.

What is polynomial regression?

Polynomial regression is a regression technique that models a curved relationship between an input feature and a continuous target by adding polynomial terms such as x², x³, and higher powers. It is useful when a straight line is too simple to represent the pattern in the data. Basic idea Simple linear regression uses: Polynomial regression can extend this to: The highest power included is called the degree of the polynomial. Simple example Suppose sales increase with advertising spend at first, but the gains become smaller at higher spending levels. A straight line may not capture that curvature well. A quadratic model such as: can represent a curved relationship. Why is it still called linear regression? Even though the relationship with x can be curved, the model is still linear in its coefficients : The model is fitting a weighted sum of transformed features such as x, x², and x³. How polynomial features are created If the original feature is: a degree 3 polynomial representation may create: These transformed features are then used by a linear regression model. Higher degree means more flexibility A higher degree polynomial can represent more complex curves. But increasing the degree too much can make the model fit noise instead of the underlying pattern, leading to overfitting. Because of this, the polynomial degree should be chosen using validation data rather than simply making it as large as possible. Polynomial regression is not the same as every nonlinear model Polynomial regression is one specific way to introduce nonlinearity by transforming features. Other models can capture nonlinear relationships without explicitly creating polynomial terms. In short, polynomial regression extends linear regression with powers of the input features so the model can represent curved relationships while remaining linear in its coefficients.

When should polynomial regression be used?

Polynomial regression should be used when the relationship between the input and a continuous target is curved, but can still be represented reasonably well using powers of the input features. It is a useful extension of linear regression when a straight line clearly underfits the data. Use it when the data shows a smooth curved pattern For example, suppose increasing advertising spend improves sales rapidly at first, but the improvement slows at higher spending levels. A straight line may miss that shape, while a quadratic model such as: can capture the curvature. Use it when the relationship is nonlinear but still fairly simple Polynomial regression works well for patterns such as: gradual upward or downward curves U shaped relationships inverted U shaped relationships relationships with one or a few smooth bends It is especially useful when you want a model that remains relatively interpretable. Use validation to choose the degree The polynomial degree controls how flexible the curve becomes. For example: degree 1 gives a straight line degree 2 allows one quadratic curve degree 3 allows more complex curvature A degree should be chosen based on validation performance rather than simply increasing it until the training data is fitted closely. Do not use a high degree just to fit every point Very high degree polynomials can follow noise in the training data and produce unstable predictions. This is a form of overfitting. Be careful with extrapolation Polynomial models can behave unpredictably outside the range of the training data because higher order terms can grow very quickly. So they are usually safer for interpolation within the observed data range than for long range extrapolation. When polynomial regression may not be the best choice Consider another model when: the relationship has many irregular changes there are many features and interactions the data contains sharp thresholds or discontinuities a tree based or other nonlinear model captures the pattern more naturally In short, use polynomial regression when a straight line model is too simple but the underlying relationship is still smooth enough to be represented by a low degree polynomial. Choose the degree with validation and avoid unnecessarily complex curves.

What is Ridge regression?

Ridge regression is a linear regression method that adds an L2 regularization penalty to discourage very large coefficient values. It is useful when ordinary linear regression is overfitting or when several input features are strongly correlated. Core idea Ordinary least squares minimizes: Ridge regression minimizes: where: bj are the feature coefficients λ controls the strength of regularization What does the penalty do? The L2 penalty makes large coefficient values more expensive. As a result, Ridge regression tends to shrink coefficients toward zero. This can make the model less sensitive to noise in the training data. What does λ control? If: Ridge regression behaves like ordinary linear regression. As λ increases: coefficients are shrunk more strongly the model becomes less flexible variance can decrease bias can increase The best value of λ is usually chosen using validation or cross validation. Why is Ridge useful with multicollinearity? When predictors contain overlapping information, ordinary linear regression coefficients can become unstable. Ridge regularization helps stabilize those coefficients by preventing any one coefficient from becoming excessively large. Does Ridge remove features? Usually, no. Ridge tends to make coefficients smaller but does not normally force them exactly to zero. So it keeps all features in the model while reducing their influence. Feature scaling matters Because the penalty depends on coefficient size, features should generally be placed on comparable scales before applying Ridge regression. Otherwise, the regularization penalty can affect features unevenly simply because they use different units. What about the intercept? The intercept is typically not regularized. The penalty is applied to the feature coefficients rather than the baseline prediction. In short, Ridge regression adds an L2 penalty to linear regression, shrinking coefficients to reduce overfitting and improve stability, especially when predictors are correlated.

What is Lasso regression?

Lasso regression is a linear regression method that adds an L1 regularization penalty to discourage large coefficient values. Its key feature is that it can shrink some coefficients exactly to zero , which means it can perform feature selection while fitting the regression model. Core idea Ordinary least squares minimizes: Lasso regression minimizes: where: bj are the feature coefficients λ controls the strength of regularization What does the L1 penalty do? The penalty discourages large coefficient values. As regularization becomes stronger, some coefficients are reduced toward zero and may become exactly zero. When a coefficient becomes zero, that feature no longer contributes to the model's prediction. Why is Lasso useful? Lasso can help: reduce overfitting simplify the model remove weak or unnecessary features improve interpretability This makes it especially useful when a dataset contains many candidate features but only some are expected to be important. What does λ control? If: Lasso behaves like ordinary linear regression. As λ increases: coefficients are penalized more strongly more coefficients may become zero the model becomes simpler too much regularization can cause underfitting The value of λ is usually selected using validation or cross validation. Feature scaling matters Because the penalty depends on coefficient size, numerical features should generally be placed on comparable scales before applying Lasso. Otherwise, features measured in different units may be penalized unevenly. What happens with correlated features? When several features contain very similar information, Lasso may keep one feature and reduce others to zero. Which feature is selected can sometimes be unstable when predictors are strongly correlated. What about the intercept? The intercept is typically not included in the L1 penalty. In short, Lasso regression adds an L1 penalty to linear regression, shrinking coefficients and potentially setting some exactly to zero, which provides both regularization and automatic feature selection.

What is Elastic Net regression?

Elastic Net regression is a regularized linear regression method that combines the L1 penalty of Lasso with the L2 penalty of Ridge regression . It is useful when you want both coefficient shrinkage and feature selection, especially when many predictors are correlated. Core idea Elastic Net minimizes a loss that includes: A common form is: where: bj are the feature coefficients the L1 term can push some coefficients exactly to zero the L2 term shrinks coefficients and improves stability Why combine Lasso and Ridge? Lasso is useful for feature selection because it can remove features by setting their coefficients to zero. Ridge is useful when predictors are strongly correlated because it tends to distribute weight more smoothly across them. Elastic Net combines both behaviors. What happens with correlated features? When several predictors contain similar information, Lasso may keep one and remove the others. Elastic Net is often more stable because the Ridge component encourages correlated predictors to share influence rather than forcing the model to choose only one. How is the balance controlled? Implementations commonly use: an overall regularization strength a mixing parameter that controls the balance between L1 and L2 For example: more L1 behavior means more coefficients may become zero more L2 behavior means stronger coefficient shrinkage without as much feature removal The best values are usually selected with validation or cross validation. Feature scaling matters Elastic Net penalizes coefficient sizes, so numerical features should generally be placed on comparable scales before fitting the model. Otherwise, features measured in different units may be penalized unevenly. When is Elastic Net useful? It is particularly useful when: there are many input features some features are irrelevant groups of predictors are highly correlated you want a simpler model without relying entirely on Lasso's feature selection behavior In short, Elastic Net combines L1 and L2 regularization so it can both shrink coefficients and remove some features while remaining more stable when predictors are correlated.

How do Ridge and Lasso differ?

Ridge and Lasso are both regularized linear regression methods, but they penalize coefficients differently. That difference changes how the two models shrink coefficients and whether they can remove features. Ridge uses L2 regularization Ridge adds a penalty based on squared coefficient values: It shrinks coefficients toward zero but usually does not make them exactly zero. So Ridge typically keeps all features in the model. Lasso uses L1 regularization Lasso adds a penalty based on absolute coefficient values: This penalty can shrink some coefficients exactly to zero. When that happens, those features are effectively removed from the model. The main practical difference Ridge: reduces the influence of features while usually retaining all of them. Lasso: can both shrink coefficients and perform automatic feature selection. What happens with correlated features? Suppose several predictors contain very similar information. Ridge usually distributes the influence across those correlated features. Lasso may keep one of them and shrink the others to zero. Because of this, Lasso's selected features can sometimes change when highly correlated predictors contain nearly the same information. When is Ridge often preferred? Ridge can be a good choice when: many features contribute some useful information predictors are strongly correlated you want stable coefficients without removing variables completely When is Lasso often preferred? Lasso can be useful when: there are many candidate features only a smaller subset is expected to matter automatic feature selection would improve interpretability Both require choosing regularization strength For both methods, λ controls how strongly coefficients are penalized. A larger λ means stronger regularization. The value is usually chosen using validation or cross validation. Feature scaling matters for both Because both penalties depend on coefficient magnitude, numerical features should generally be placed on comparable scales before fitting Ridge or Lasso. What if you want characteristics of both? Elastic Net combines L1 and L2 regularization, providing a middle ground between Lasso's feature selection and Ridge's stability. In short, Ridge uses L2 regularization to shrink coefficients while usually keeping every feature, whereas Lasso uses L1 regularization and can shrink some coefficients exactly to zero, effectively selecting features.

How can regression models handle nonlinear relationships?

Regression models can handle nonlinear relationships by transforming the inputs or by using model families that are not restricted to a straight line relationship. 1. Add polynomial features A linear regression model can include powers of a feature such as: For example: This allows the prediction to follow a curved pattern while the model remains linear in its coefficients. 2. Transform the input features Sometimes the relationship becomes easier to model after transforming a feature. Common transformations include: logarithm: log(x) square root: √x reciprocal: 1/x For example, if the target changes rapidly at small values of x but more slowly later, a logarithmic transformation may better represent the pattern. 3. Add interaction features A feature's effect may depend on another feature. An interaction term can represent this: For example, the effect of advertising spend may depend on the product's price level. 4. Use piecewise functions or splines Splines divide the input range into regions and fit smooth functions across them. They can capture complex curves without requiring one very high degree polynomial across the entire dataset. 5. Use nonlinear regression algorithms Some regression algorithms naturally capture nonlinear relationships, including: decision trees random forests gradient boosted trees kernel based regression methods These models can learn nonlinear patterns without explicitly creating polynomial terms. Choosing the right approach A more flexible model is not automatically better. If the model is too simple, it may underfit. If it is too flexible, it may fit noise and overfit. The appropriate transformation, polynomial degree, or model type should therefore be chosen using validation performance. Interpretability also matters Feature transformations and low degree polynomial regression are often easier to explain than more complex nonlinear models. So the choice should consider both predictive performance and how much interpretability the problem requires. In short, regression models handle nonlinear relationships by transforming features, adding polynomial or interaction terms, using splines, or switching to nonlinear regression algorithms. The right level of flexibility should be chosen using validation rather than training fit alone.

How do you evaluate a regression model?

A regression model is evaluated by checking how accurately it predicts continuous values on data it did not train on. The goal is not simply to fit the training data well, but to verify that the model generalizes to new observations. 1. Evaluate on unseen data Use a validation or test dataset that was not used to fit the model. If performance is strong on training data but much worse on unseen data, the model may be overfitting. 2. Measure prediction error Common regression metrics include: MAE — average absolute prediction error MSE — average squared prediction error RMSE — square root of MSE, expressed in the target's units R² — how much of the target's variation is explained by the model relative to a simple mean baseline No single metric is best for every problem. For example, RMSE penalizes large errors more strongly than MAE, while MAE is often easier to interpret directly. 3. Compare against a baseline A regression model should outperform a simple baseline. For example, a basic baseline might predict the average target value for every observation. If a complex model cannot beat that baseline on unseen data, the additional complexity may not be useful. 4. Analyze residuals Metrics summarize performance, but residuals can reveal how the model is failing. Look for: systematic curves or patterns changing residual spread unusually large errors time dependent patterns Clear structure in the residuals can indicate that the model is missing an important relationship. 5. Check performance across important groups An acceptable overall score can hide poor performance for certain ranges or segments. For example, a house price model might work well for mid priced homes but make large errors for expensive properties. Evaluate important subgroups when the application requires it. 6. Use cross validation when appropriate Cross validation evaluates the model across multiple train validation splits instead of relying on a single split. This gives a more stable estimate of expected performance when the dataset and problem structure allow random resampling. Time dependent data should instead use a splitting strategy that preserves chronological order. 7. Judge the errors in business or domain context A metric value is useful only when interpreted in the context of the problem. For example, an MAE of ₹5,000 may be excellent for predicting house prices but unacceptable for predicting a much smaller financial amount. In short, evaluate a regression model on unseen data using appropriate error metrics, compare it with a simple baseline, inspect residuals, and confirm that its errors are acceptable for the real world problem.

What is Mean Absolute Error (MAE)?

Mean Absolute Error (MAE) measures the average absolute difference between a regression model's predictions and the actual target values. It answers a simple question: On average, how far are the predictions from the true values? Formula where: yi is the actual value ŷi is the predicted value n is the number of observations Simple example Suppose the actual values are: and the predictions are: The absolute errors are: So: This means the model's predictions are off by 3 units on average . Why use absolute values? Prediction errors can be positive or negative. If we averaged the raw errors directly, overpredictions and underpredictions could cancel each other out. Taking the absolute value prevents that cancellation. MAE is easy to interpret MAE is expressed in the same unit as the target variable . For example: house price MAE of ₹50,000 means predictions differ from actual prices by about ₹50,000 on average temperature MAE of 2°C means predictions are off by about 2°C on average How does MAE treat large errors? Every error contributes in direct proportion to its size. An error of 10 contributes twice as much as an error of 5. Because errors are not squared, MAE is generally less dominated by very large errors than squared error metrics such as MSE. Is lower MAE better? Yes. means every prediction is exactly correct. In general, a lower MAE indicates smaller average prediction errors, but whether a particular value is good depends on the scale and business context of the target. In short, MAE measures the average absolute prediction error in the same units as the target, making it one of the easiest regression metrics to interpret.