When should outliers be removed?

Outliers should be removed only when there is a strong reason to believe they are incorrect, irrelevant, or harmful to the intended machine learning task.

Being unusual is not enough by itself to justify deletion.

Remove outliers that are clear data errors

Removal is often appropriate when an extreme value is impossible or obviously incorrect.

For example:

• an age of 250 years
• a negative quantity where negative values are impossible
• a sensor reading caused by a known device failure
• a value entered in the wrong unit

If the correct value cannot be recovered, removing the affected record may be reasonable.

Remove records that belong to the wrong population

An observation may be valid but irrelevant to the problem being modeled.

For example, a dataset intended to model ordinary retail customers may accidentally contain internal test accounts or system-generated transactions.

Those records can behave like outliers even though their values are technically valid.

Consider removal when a small number of extreme points distort the model

Some algorithms are very sensitive to extreme values.

If a few unusual observations have a large and unjustified influence on the fitted model, removing them may improve stability.

However, first confirm that those observations are not important real-world cases.

Do not remove outliers only because a statistical rule flags them

A point outside an IQR boundary or with a large z-score is only a candidate outlier.

It may still represent a valid observation.

For example, very large transactions may be rare but legitimate.

Check the effect of removal

Before permanently removing outliers, compare the dataset and model with and without them.

Check whether removal:

• improves validation performance
• changes important relationships
• disproportionately removes a particular group
• makes the training data less representative of production data

Use alternatives when possible

Instead of deleting valid extreme values, you may consider:

• transforming the feature
• capping extreme values
• using robust scaling
• choosing models or loss functions that are less sensitive to outliers

In short, remove outliers when they are confirmed errors, irrelevant records, or unjustified extreme points that distort the model. Valid rare cases should usually be retained rather than deleted simply because they are unusual.