What is feature selection in machine learning?

Feature selection is the process of choosing the most useful input features for a machine learning model and removing features that are irrelevant, redundant, or unnecessarily noisy.

The goal is to keep the information that helps prediction while reducing unnecessary complexity.

Simple example

Suppose a dataset contains 100 input features, but only 20 of them provide meaningful predictive information.

Feature selection tries to identify and keep those useful features instead of training the model on all 100.

Why remove unnecessary features?

Too many weak or redundant features can:

• make the model harder to train
• increase computation and memory usage
• introduce noise
• make interpretation more difficult
• increase the risk of overfitting in some situations

Feature selection can create a cleaner input space.

Feature selection keeps original features

Feature selection is different from feature extraction.

Feature selection chooses from the existing columns.

For example:

Original features: A, B, C, D, E
Selected features: A, C, E

The selected features remain unchanged.

Feature extraction, by contrast, can transform the original features into a new representation.

Main approaches

Feature-selection methods are commonly grouped into:

• Filter methods: rank or select features using statistical properties before training the final model
• Wrapper methods: evaluate different feature subsets using model performance
• Embedded methods: perform feature selection as part of model training

Each approach has different trade-offs in speed, computational cost, and model dependence.

Selection should use training data only

Any feature-selection decision that uses information from the target or model performance should be made using training data, typically inside the validation process.

Selecting features using the full dataset before evaluation can leak information from validation or test data.

More features are not always better

Adding every available column can sometimes hurt rather than help.

A smaller set of strong, relevant features can produce a model that is faster, easier to understand, and sometimes more accurate.

In short, feature selection identifies and keeps the most useful existing features so a machine learning model can focus on relevant information while reducing unnecessary complexity.