What is correlation-based feature selection?

Correlation-based feature selection uses correlation values to identify features that are strongly related to the target or highly redundant with other features.

It is a common filter-based approach for numerical data.

Correlation with the target

A feature with a strong correlation to a numerical target may contain useful predictive information.

For example:

Feature A correlation with target = 0.82
Feature B correlation with target = 0.05

Feature A appears to have a much stronger linear relationship with the target than Feature B.

Correlation between features

Correlation can also be used to detect redundancy.

If two features are highly correlated with each other, such as:

Feature X vs Feature Y = 0.97

they may be carrying almost the same information.

In that case, one of them may be removed if it does not add additional value.

Positive and negative correlation both matter

A strong negative correlation can be just as informative as a strong positive correlation.

For example:

correlation = -0.85

still represents a strong linear relationship.

Feature selection therefore often considers the absolute correlation value.

Main advantage

Correlation-based selection is:

• fast
• simple to calculate
• easy to interpret
• useful as an initial screening method

It works well when linear relationships are important.

Main limitation

Correlation mainly measures linear relationships.

A feature can have very low correlation with the target and still be highly predictive through a nonlinear relationship.

For example, if the target depends on x², the raw feature x may have weak linear correlation even though it is important.

Correlation does not prove causation

A strong correlation means two variables move together statistically.

It does not prove that one variable causes the other.

Avoid leakage

If correlation with the target is used for feature selection, calculate it using training data only.

Using the entire dataset before validation can leak information from validation or test data.

In short, correlation-based feature selection uses correlation to keep features that show useful linear relationships with the target and to remove highly redundant features, but it can miss important nonlinear relationships.