When should outliers be retained?

Outliers should be retained when they are valid observations that represent real situations the model may encounter.

An observation should not be removed only because it is rare or far from the average.

Retain valid rare cases

If an extreme value is correct and belongs to the population being modeled, it may contain important information.

For example:

• a very large but legitimate purchase
• an unusually high-value customer
• a rare equipment failure
• an uncommon medical case

Removing these records can make the training data less representative of reality.

Retain outliers when rare events are the prediction target

In some machine learning problems, unusual observations are exactly what matters.

Examples include:

• fraud detection
• anomaly detection
• fault detection
• cybersecurity monitoring

Removing unusual records from these datasets could remove the cases the model is supposed to identify.

Retain them when the distribution is naturally skewed

Some real-world variables naturally contain a long tail of extreme values.

Examples include:

• income
• transaction amounts
• website traffic
• insurance claims

In such datasets, extreme observations may be part of the true distribution rather than errors.

Retain them when they may occur in production

The training data should reflect the conditions the model will face after deployment.

If valid extreme values can appear in production, removing all of them from training can leave the model poorly prepared for those cases.

Use robust methods instead of deletion when appropriate

If valid outliers strongly affect the model, consider alternatives such as:

• transforming the feature
• using robust scaling
• capping only when justified
• choosing models or loss functions that are less sensitive to extreme values

This can reduce their unwanted influence without discarding real information.

Investigate first

Before retaining an outlier, confirm that:

• the value is valid
• it was measured or recorded correctly
• it belongs to the population of interest
• it is relevant to the machine learning task

In short, retain outliers when they are valid, relevant, and representative of situations the model may encounter. Rare does not mean incorrect, and removing genuine extreme cases can make a model less useful in the real world.