Machine Learning
Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.
What is simple linear regression?
Simple linear regression is a regression method that models the relationship between one input feature and one continuous target variable using a straight line. Basic idea Suppose you want to predict house price using only house size. Simple linear regression tries to fit a line such as: The line is chosen so that its predictions are as close as possible to the actual target values. The equation A simple linear regression model is commonly written as: where: x is the input feature y is the predicted target b0 is the intercept b1 is the slope or coefficient What does the slope mean? The slope tells us how much the predicted target changes when the input increases by one unit. For example: If size increases by one unit, the predicted price increases by 5,000, assuming the model uses that unit of size. What does the intercept mean? The intercept is the predicted value of the target when the input feature is zero. Whether that value is meaningful depends on the problem. Why is it called simple? It is called simple linear regression because it uses only one input feature. If several input features are used, the model is called multiple linear regression . What relationship does it assume? Simple linear regression assumes that the expected relationship between the input and target can be reasonably represented by a straight line. If the real relationship is strongly curved, a simple linear model may not fit the data well. Example use cases Simple linear regression can be used for relationships such as: advertising spend → sales house size → house price study hours → exam score temperature → electricity usage when a single feature provides a useful approximately linear signal. In short, simple linear regression predicts a continuous target from one input feature by fitting a straight line relationship between them.
How does linear regression work?
Linear regression works by finding the linear relationship between input features and a continuous target that best fits the training data. It learns coefficients for the input features and then uses those coefficients to make numerical predictions. Basic idea With one input feature, the model has the form: where: ŷ is the predicted value b0 is the intercept b1 is the coefficient or slope x is the input feature The model tries to find values for b0 and b1 that make the predicted values close to the actual targets. How training works During training: the model starts with candidate coefficient values it generates predictions for the training examples it measures the difference between each prediction and the actual target it adjusts the coefficients to reduce those errors For ordinary linear regression, the coefficients are commonly chosen to minimize the sum of squared residuals . What is a residual? A residual is the difference between an observed target and the model's prediction: For example: The model tries to find a line that keeps these residuals as small as possible overall. With multiple features When several input features are used, the model becomes: Each coefficient represents how the prediction changes with that feature while the other included features are held constant. Making a prediction After the coefficients have been learned, a new observation is passed into the equation. For example: The result is the model's numerical prediction. Why it is called linear Linear regression is linear in its coefficients: the prediction is a weighted sum of the input features plus an intercept. If the underlying relationship is strongly nonlinear and the features do not represent that shape, a basic linear model may underfit the data. In short, linear regression learns coefficients that define a best fitting linear relationship between the inputs and target, then uses that learned equation to predict numerical values for new observations.
What is the equation of a linear regression model?
The equation of a linear regression model represents the target as a weighted combination of the input features plus an intercept. Simple linear regression With one input feature, the prediction equation is: where: ŷ is the predicted target value b0 is the intercept b1 is the coefficient or slope x is the input feature Multiple linear regression With several input features, the equation becomes: Each feature has its own coefficient. For example: What does the intercept do? The intercept b0 is the model's predicted value when all input features are zero. Whether that value has a practical interpretation depends on the problem. What do the coefficients do? Each coefficient controls how much the prediction changes when its corresponding feature changes by one unit, while the other included features are held constant. For example: If x increases by 1, the predicted value increases by 4. Statistical form of the model When describing the observed target rather than only the prediction, linear regression is often written as: Here, ε represents the error term: the part of the observed target that is not explained by the linear relationship. Prediction form vs observed value form Prediction equation: uses ŷ and does not include the unknown error term. Statistical model equation: uses y and includes ε to represent unexplained variation. Why is the model called linear? The prediction is a linear combination of the coefficients and input features: In short, a linear regression model predicts a numerical target by adding an intercept to the weighted sum of its input features.
What do coefficients mean in linear regression?
Coefficients in linear regression represent how much the model’s predicted target changes when an input feature changes by one unit. In a multiple linear regression model, each coefficient describes the effect of its feature while the other included features are held constant. Basic equation A linear regression model can be written as: where each b value is a coefficient. Positive coefficient A positive coefficient means the predicted target tends to increase as the feature increases. For example: If the advertising spend feature increases by one unit, the predicted sales value increases by 4 units. Negative coefficient A negative coefficient means the predicted target tends to decrease as the feature increases. For example: If distance increases by one unit, the predicted house price decreases by 20,000 units, assuming the other included features remain unchanged. The magnitude shows the change per unit A larger absolute coefficient means a larger change in the prediction for a one unit change in that feature. However, coefficient magnitudes should not be compared blindly when features use different units or scales. For example, a coefficient for income measured in rupees may look much smaller than a coefficient for age measured in years simply because the units are very different. Coefficients in multiple regression In a model with several features, a coefficient should be interpreted as: This conditional interpretation is important. Categorical variables can also have coefficients After categorical variables are encoded, their coefficients usually represent differences relative to a reference category. For example, with one hot encoding, a category coefficient can indicate how much the prediction differs from the omitted baseline category. Coefficients do not prove causation A positive or negative coefficient describes an association learned by the model. It does not automatically mean that changing the feature will cause the target to change. Multicollinearity can make coefficients unstable If predictors contain strongly overlapping information, individual coefficient values can become unstable or difficult to interpret. This is why multicollinearity matters especially when coefficient interpretation is important. In short, a linear regression coefficient tells you how the model’s predicted target changes for a one unit change in a feature, with its sign showing direction and its magnitude showing the size of that modeled change.
What is the intercept in linear regression?
The intercept in linear regression is the model’s predicted target value when all input features are equal to zero. In the regression equation: b0 is the intercept. Simple example Suppose a model is: The intercept is: This means the model predicts a salary of 25,000 when years of experience equals zero. What does the intercept do? The intercept shifts the regression line or regression surface up or down. Without an intercept, the model is forced to pass through zero. For example: forces the prediction to be zero whenever x = 0. That constraint may be unrealistic for many real world problems. Is the intercept always meaningful? Not necessarily. If zero is outside the realistic range of a feature, the intercept may have little practical interpretation. For example, if a house size dataset only contains homes between 800 and 3,000 square feet, the predicted price for a house of zero square feet is not a meaningful real world scenario. Even so, the intercept can still help the model fit the observed data correctly. Multiple linear regression With several features, the intercept represents the predicted target when all included features are zero at the same time . Whether that scenario is realistic depends on the variables and how they are represented. Centering changes the interpretation If numerical features are centered around their means, zero represents an average feature value rather than the original zero. In that case, the intercept can be interpreted as the predicted target for an observation with average values for those centered features. Should the intercept be removed? Usually, linear regression models include an intercept by default. Removing it is appropriate only when there is a strong theoretical reason that the relationship must pass through the origin. In short, the intercept is the baseline prediction of a linear regression model when all input features are zero. It helps position the regression line correctly, even when the zero input case itself is not practically meaningful.
What is multiple linear regression?
Multiple linear regression predicts a continuous numerical target using two or more input features . It extends simple linear regression, which uses only one input feature. Basic equation A multiple linear regression model is written as: where: ŷ is the predicted target b0 is the intercept b1, b2, ... bn are coefficients x1, x2, ... xn are input features Simple example Suppose we want to predict house price using: house size number of rooms distance from the city center The model might look like: Each feature contributes to the final prediction. How are the coefficients interpreted? Each coefficient represents the expected change in the predicted target for a one unit change in that feature, while the other included features are held constant . For example, if the coefficient for house size is positive, larger houses are associated with higher predicted prices after accounting for the other features in the model. Why use multiple features? Many real world outcomes depend on several factors at the same time. For example, house price may depend on size, location, age, and number of rooms rather than only one variable. Multiple linear regression allows those factors to be modeled together. It is still a linear model Even with many features, the prediction is still a weighted sum of the inputs plus an intercept. The model is therefore linear in its coefficients. More features are not automatically better Adding irrelevant or highly redundant features can make the model harder to interpret and may introduce problems such as multicollinearity. Feature choice should therefore be based on domain knowledge and validation. In short, multiple linear regression predicts a numerical target using several input features, assigning a coefficient to each feature so their combined linear effect can be modeled.
What assumptions does linear regression make?
Linear regression relies on several assumptions about the relationship between the features, target, and errors. These assumptions matter because violating them can make coefficients, uncertainty estimates, or statistical conclusions unreliable. 1. Linearity The expected target should be a linear combination of the included features. In simple form: This does not mean every raw relationship in the real world must be perfectly straight, but the model must represent the relationship adequately through the features it uses. 2. Independent observations or errors Observations should not have error terms that are systematically dependent on one another. For example, measurements collected over time may have correlated errors if nearby time points influence each other. This issue is often called autocorrelation. 3. Constant error variance The variance of the residuals should be approximately constant across the range of predictions. This assumption is called homoscedasticity . If the error spread grows or shrinks systematically, the data shows heteroscedasticity . 4. No perfect multicollinearity One predictor should not be an exact linear combination of other predictors. For example: would create perfect multicollinearity. Strong near multicollinearity may also make individual coefficients unstable. 5. Errors should have zero conditional mean After accounting for the included predictors, the remaining error should not systematically depend on those predictors. In practical terms, the model should not be missing important information that is both related to the included features and the target in a way that biases the coefficients. This is closely related to the idea of exogeneity . 6. Normality of errors for classical inference Residuals do not need to be perfectly normal for linear regression to produce predictions or ordinary least squares coefficient estimates. However, approximate normality of the errors is useful for classical hypothesis tests and confidence intervals, especially in smaller samples. How are these assumptions checked? Common diagnostic approaches include: residual plots plots of residuals against fitted values checking correlations or VIF for multicollinearity examining residual distributions checking residual dependence for ordered or time based data Residual analysis is covered in the next articles. In short, linear regression works best when the relationship is adequately linear, errors are independent with roughly constant variance and zero conditional mean, predictors do not have perfect multicollinearity, and error normality is reasonable when classical statistical inference is required.
What is a residual in regression?
A residual in regression is the difference between an observed target value and the value predicted by the regression model. It tells us how far a prediction is from the actual value for a particular observation. Residual formula or: where: y is the actual target value ŷ is the predicted target value e is the residual Simple example Suppose the actual house price is: but the model predicts: Then: The positive residual means the model underpredicted the actual value by ₹5 lakh. What does a negative residual mean? If: then: A negative residual means the model overpredicted the target. What does a residual of zero mean? A residual of zero means: So the model predicted that observation exactly. Residuals are available after fitting the model Once a regression model makes predictions on known observations, a residual can be calculated for each one. These residuals help us understand where and how the model is making errors. Residual vs error term The theoretical regression model contains an unknown error term , representing the true unexplained part of the target. A residual is the observed estimate of that error after the model has been fitted. Why residuals matter Residuals are useful for checking whether the regression model is behaving as expected. Patterns in residuals can reveal problems such as nonlinearity, changing error variance, unusual observations, or other model misspecification. How residuals are analyzed is covered in the next article. In short, a residual is the difference between an actual target value and the model's prediction for that observation.
How do you analyze residuals?
Residual analysis checks the errors left by a regression model to see whether the model is capturing the important structure in the data. A well specified regression model usually produces residuals that look like random noise rather than showing clear patterns. Start with residuals vs fitted values Plot: Ideally, the residuals should be scattered roughly randomly around zero. What patterns can reveal Curved pattern: the relationship may not be adequately linear. Funnel shape: the residual variance may change with the prediction, suggesting heteroscedasticity. Clusters or bands: the model may be missing an important grouping variable or structure. Large isolated residuals: some observations may be unusual or poorly predicted. Check whether residuals are centered around zero Residuals should not be systematically positive or negative across a region of the data. If they are, the model may be consistently underpredicting or overpredicting in that region. Examine residuals against individual predictors Plotting residuals against important input features can reveal relationships that the fitted model did not capture. For example, a curved residual pattern against a feature may suggest that a transformation or nonlinear term is needed. Check residual spread The vertical spread of residuals should be reasonably similar across fitted values. If the spread becomes much wider or narrower, the constant variance assumption may be violated. Check dependence for ordered or time based data For time ordered observations, inspect residuals in chronological order. Long runs of positive or negative residuals or repeating patterns can indicate autocorrelation or missing time related structure. Examine the residual distribution when inference matters A histogram or Q Q plot can help assess whether residuals are approximately normally distributed. Normality is mainly important for classical confidence intervals and hypothesis tests, especially with smaller samples. Investigate unusual observations Very large residuals may indicate: data quality problems unusual cases missing variables model misspecification Large residuals should be investigated rather than automatically removed. Residual analysis is diagnostic, not a single pass/fail test No real dataset will produce perfectly random residuals. The goal is to look for systematic patterns that suggest the model can be improved. In short, analyze residuals by checking whether they are randomly scattered around zero with roughly constant spread and no systematic pattern. Curves, funnels, dependence, or extreme residuals can reveal problems with the regression model.
What is ordinary least squares (OLS)?
Ordinary Least Squares (OLS) is a method used to estimate the coefficients of a linear regression model. It chooses the regression line or regression plane that makes the model's prediction errors as small as possible according to the sum of squared residuals . Core idea For each training observation: OLS squares those residuals and adds them together: The coefficients are chosen so that this total is as small as possible. Simple example Suppose the residuals from a candidate regression line are: The squared residuals are: So the total squared error is: OLS searches for coefficient values that produce the smallest possible total squared error. Why are residuals squared? Squaring has two useful effects: positive and negative residuals cannot cancel each other out larger errors receive a stronger penalty than smaller errors This creates a clear optimization objective for fitting the regression model. What does OLS estimate? For a model such as: OLS estimates: the intercept b0 the feature coefficients b1, b2, ... bn OLS is an estimation method OLS is not a separate prediction problem from linear regression. Linear regression defines the form of the model, while OLS is one common method for estimating its coefficients. Why is it called ordinary? The word ordinary distinguishes standard least squares from related approaches such as weighted least squares or generalized least squares, which modify how errors are handled. OLS can be sensitive to large errors Because residuals are squared, observations with very large residuals can have a strong influence on the fitted model. This is one reason outliers and residual diagnostics are important when using OLS. In short, OLS fits a linear regression model by choosing the coefficients that minimize the sum of squared differences between actual and predicted values.