What is Lasso regression?

Lasso regression is a linear regression method that adds an L1 regularization penalty to discourage large coefficient values.

Its key feature is that it can shrink some coefficients exactly to zero, which means it can perform feature selection while fitting the regression model.

Core idea

Ordinary least squares minimizes:

Sum of Squared Residuals

Lasso regression minimizes:

Sum of Squared Residuals + λ × Σ|bj|

where:

• bj are the feature coefficients
• λ controls the strength of regularization

What does the L1 penalty do?

The penalty discourages large coefficient values.

As regularization becomes stronger, some coefficients are reduced toward zero and may become exactly zero.

When a coefficient becomes zero, that feature no longer contributes to the model's prediction.

Why is Lasso useful?

Lasso can help:

• reduce overfitting
• simplify the model
• remove weak or unnecessary features
• improve interpretability

This makes it especially useful when a dataset contains many candidate features but only some are expected to be important.

What does λ control?

If:

λ = 0

Lasso behaves like ordinary linear regression.

As λ increases:

• coefficients are penalized more strongly
• more coefficients may become zero
• the model becomes simpler
• too much regularization can cause underfitting

The value of λ is usually selected using validation or cross-validation.

Feature scaling matters

Because the penalty depends on coefficient size, numerical features should generally be placed on comparable scales before applying Lasso.

Otherwise, features measured in different units may be penalized unevenly.

What happens with correlated features?

When several features contain very similar information, Lasso may keep one feature and reduce others to zero.

Which feature is selected can sometimes be unstable when predictors are strongly correlated.

What about the intercept?

The intercept is typically not included in the L1 penalty.

In short, Lasso regression adds an L1 penalty to linear regression, shrinking coefficients and potentially setting some exactly to zero, which provides both regularization and automatic feature selection.