What is recursive feature elimination (RFE)?

Recursive Feature Elimination (RFE) is a wrapper-based feature-selection method that repeatedly trains a model and removes the least important features.

The process continues until the desired number of features remains.

How RFE works

RFE typically follows these steps:

• train a model using all available features
• calculate feature importance or model coefficients
• remove the least important feature or group of features
• train the model again using the remaining features
• repeat until the required number of features is reached

Simple example

Suppose a dataset starts with:

A, B, C, D, E

After training, the model considers D the least useful feature.

RFE removes it:

A, B, C, E

The model is trained again, another weak feature is removed, and the process continues.

Why RFE can be useful

RFE evaluates features in the context of a real model.

This means the selected feature set is based on how features contribute to the chosen algorithm rather than only on a simple statistical score.

RFE depends on the model

The underlying model must provide some way to rank features, such as:

• model coefficients
• feature importance scores

For example, linear models can use coefficient magnitudes, while some tree-based models provide feature-importance measures.

Main limitation

RFE can be computationally expensive because the model is trained repeatedly.

The cost becomes more significant when:

• the dataset has many features
• the underlying model is slow to train
• cross-validation is also used

Choosing the number of features

The number of features to keep can be chosen manually or evaluated using validation.

A common extension, RFECV, combines RFE with cross-validation to help choose the number of selected features.

Avoid data leakage

RFE should be performed inside the training and validation process.

Running RFE once on the full dataset before evaluating the model can leak information from validation or test data.

In short, RFE repeatedly trains a model, removes the least important features, and continues until a smaller feature set remains. It is model-aware and often effective, but can be computationally expensive.