What are embedded feature-selection methods?

Embedded feature-selection methods select important features as part of the model-training process itself.

Unlike filter methods, which score features before model training, and wrapper methods, which repeatedly test different feature subsets, embedded methods make feature-selection decisions while the model is being fitted.

How embedded methods work

The learning algorithm includes a mechanism that reduces the influence of weak features or identifies which features contribute most to prediction.

After training, features with little contribution can be removed or given very low importance.

L1 regularization is a common example

Models such as Lasso regression use L1 regularization.

L1 regularization can push some feature coefficients exactly to zero.

For example:

Feature A coefficient = 1.8
Feature B coefficient = 0
Feature C coefficient = -0.7

Feature B contributes nothing to the fitted model and can effectively be treated as unselected.

Tree-based models can also provide embedded selection

Decision trees and tree-based ensembles choose features while creating splits.

Features that repeatedly produce useful splits may receive higher importance, while features that are rarely or never used may be candidates for removal.

Why embedded methods are useful

They often provide a good balance between filter and wrapper approaches because they:

• consider the behavior of the actual model
• can capture relationships that simple statistical filters may miss
• usually require fewer repeated model fits than wrapper methods
• can perform selection and training in one process

They are model-dependent

The selected features depend on the algorithm.

For example, features selected by Lasso may differ from those considered important by a tree-based model.

There is therefore no single embedded feature set that is automatically best for every model.

Regularization strength matters

For methods such as Lasso, the amount of regularization affects how aggressively features are removed.

Too little regularization may retain many weak features, while too much may remove useful ones.

This setting should be chosen using validation.

Avoid data leakage

Embedded feature selection must be performed inside the training process.

When cross-validation is used, feature selection should be repeated within each training fold rather than selecting features once using the full dataset.

In short, embedded methods perform feature selection during model training. They are usually more model-aware than filter methods and more efficient than exhaustive wrapper methods, but the selected features depend on the algorithm and its settings.