What is ordinary least squares (OLS)?

Ordinary Least Squares (OLS) is a method used to estimate the coefficients of a linear regression model.

It chooses the regression line or regression plane that makes the model's prediction errors as small as possible according to the sum of squared residuals.

Core idea

For each training observation:

Residual = Actual value - Predicted value

OLS squares those residuals and adds them together:

Sum of Squared Residuals = Σ(y - ŷ)²

The coefficients are chosen so that this total is as small as possible.

Simple example

Suppose the residuals from a candidate regression line are:

2, -3, 1

The squared residuals are:

4, 9, 1

So the total squared error is:

4 + 9 + 1 = 14

OLS searches for coefficient values that produce the smallest possible total squared error.

Why are residuals squared?

Squaring has two useful effects:

• positive and negative residuals cannot cancel each other out
• larger errors receive a stronger penalty than smaller errors

This creates a clear optimization objective for fitting the regression model.

What does OLS estimate?

For a model such as:

ŷ = b0 + b1x1 + b2x2 + ... + bnxn

OLS estimates:

• the intercept b0
• the feature coefficients b1, b2, ... bn

OLS is an estimation method

OLS is not a separate prediction problem from linear regression.

Linear regression defines the form of the model, while OLS is one common method for estimating its coefficients.

Why is it called ordinary?

The word ordinary distinguishes standard least squares from related approaches such as weighted least squares or generalized least squares, which modify how errors are handled.

OLS can be sensitive to large errors

Because residuals are squared, observations with very large residuals can have a strong influence on the fitted model.

This is one reason outliers and residual diagnostics are important when using OLS.

In short, OLS fits a linear regression model by choosing the coefficients that minimize the sum of squared differences between actual and predicted values.