What are wrapper methods for feature selection?

Wrapper methods are feature-selection techniques that evaluate different subsets of features by training a machine learning model and measuring how well each subset performs.

Instead of scoring features only with a statistical test, wrapper methods use actual model performance to decide which features to keep.

How wrapper methods work

A wrapper method repeatedly:

• chooses a subset of features
• trains a model using that subset
• evaluates the model on validation data
• compares the result with other subsets

The feature subset that gives the best validation performance is preferred.

Common wrapper approaches

Typical strategies include:

• Forward selection: start with no features and add useful features one at a time
• Backward elimination: start with all features and remove the least useful ones
• Recursive Feature Elimination (RFE): repeatedly train a model and remove weaker features

RFE is covered separately later in this section.

Why wrapper methods can be effective

Because they evaluate features using a real model, wrapper methods can capture situations where:

• a feature is useful only when combined with another feature
• two individually strong features become redundant together
• the best feature subset depends on the chosen algorithm

This makes them more model-aware than filter methods.

Main limitation: computational cost

Wrapper methods may require training the model many times.

If a dataset has many features, the number of possible feature combinations can become extremely large.

This can make wrapper methods:

• slow
• computationally expensive
• difficult to use on very high-dimensional datasets

They can overfit the validation process

If too many feature subsets are repeatedly tested on the same validation data, the selected subset may become overly tuned to that validation set.

Cross-validation is often used to make the evaluation more reliable.

Wrapper methods are model-dependent

A feature subset that works well for one algorithm may not be the best subset for another.

This is different from filter methods, which usually select features independently of the final model.

Avoid data leakage

Feature selection must happen inside the training and validation process.

Using the full dataset to choose features before evaluating the model can leak information from validation or test data.

In short, wrapper methods select features by repeatedly training and evaluating a model on different feature subsets. They can capture feature interactions well, but they are usually more computationally expensive than filter methods.