How does least squares estimate regression coefficients?

Least squares estimates regression coefficients by choosing the coefficient values that minimize the total squared difference between the actual target values and the model's predictions.

Start with the regression equation

For simple linear regression:

ŷ = b0 + b1x

where:

• b0 is the intercept
• b1 is the slope
• x is the input feature
• ŷ is the predicted target

Calculate the residuals

For each training observation:

Residual = y - ŷ

Substituting the regression equation:

Residual = y - (b0 + b1x)

Square and sum the residuals

Least squares defines the objective:

Σ(y - ŷ)²

or:

Σ[y - (b0 + b1x)]²

The goal is to find the values of b0 and b1 that make this quantity as small as possible.

How are the best coefficients found?

Mathematically, the squared-error function can be minimized by finding where its slope with respect to the coefficients becomes zero.

For simple linear regression, this produces:

b1 = Σ[(xi - x̄)(yi - ȳ)] / Σ[(xi - x̄)²]

and:

b0 = ȳ - b1x̄

where x̄ and ȳ are the mean values of the input and target.

Intuition behind the slope

The slope becomes larger when x and y tend to increase together.

It becomes negative when larger values of x tend to be associated with smaller values of y.

The denominator measures how much the input feature varies, while the numerator measures how the input and target vary together.

With multiple features

For multiple linear regression:

ŷ = b0 + b1x1 + b2x2 + ... + bnxn

the same principle applies: choose all coefficients together so that the total squared residual is minimized.

This can be solved using linear algebra or numerical optimization methods.

Why this gives the best-fitting line

Among all possible coefficient combinations, least squares selects the one with the smallest total squared training error.

This does not mean every prediction is exact. Some residuals remain, but their squared total is as small as possible for that linear model.

In short, least squares estimates regression coefficients by adjusting them until the sum of squared residuals reaches its minimum.