What is Ridge regression?

Ridge regression is a linear regression method that adds an L2 regularization penalty to discourage very large coefficient values.

It is useful when ordinary linear regression is overfitting or when several input features are strongly correlated.

Core idea

Ordinary least squares minimizes:

Sum of Squared Residuals

Ridge regression minimizes:

Sum of Squared Residuals + λ × Σ(bj²)

where:

• bj are the feature coefficients
• λ controls the strength of regularization

What does the penalty do?

The L2 penalty makes large coefficient values more expensive.

As a result, Ridge regression tends to shrink coefficients toward zero.

This can make the model less sensitive to noise in the training data.

What does λ control?

If:

λ = 0

Ridge regression behaves like ordinary linear regression.

As λ increases:

• coefficients are shrunk more strongly
• the model becomes less flexible
• variance can decrease
• bias can increase

The best value of λ is usually chosen using validation or cross-validation.

Why is Ridge useful with multicollinearity?

When predictors contain overlapping information, ordinary linear regression coefficients can become unstable.

Ridge regularization helps stabilize those coefficients by preventing any one coefficient from becoming excessively large.

Does Ridge remove features?

Usually, no.

Ridge tends to make coefficients smaller but does not normally force them exactly to zero.

So it keeps all features in the model while reducing their influence.

Feature scaling matters

Because the penalty depends on coefficient size, features should generally be placed on comparable scales before applying Ridge regression.

Otherwise, the regularization penalty can affect features unevenly simply because they use different units.

What about the intercept?

The intercept is typically not regularized.

The penalty is applied to the feature coefficients rather than the baseline prediction.

In short, Ridge regression adds an L2 penalty to linear regression, shrinking coefficients to reduce overfitting and improve stability, especially when predictors are correlated.