How should duplicate records be handled?

Duplicate records should be handled only after confirming that they are truly redundant and not legitimate repeated observations.

Removing every repeated-looking row automatically can delete useful information.

First determine whether the duplicate is real

Ask what one row represents.

For example:

• two identical customer profiles may be accidental duplicates
• two identical product purchases may be legitimate separate transactions
• repeated sensor readings may be valid observations

The business meaning of the data should guide the decision.

Remove exact accidental duplicates

If two or more rows represent the same real-world observation and were duplicated because of data collection, import, or processing errors, keep one valid record and remove the redundant copies.

This prevents some observations from being counted more than once.

Resolve conflicting duplicates carefully

Sometimes records refer to the same entity but contain different values.

For example, two rows with the same customer ID may contain different addresses or timestamps.

In that case, do not simply delete one row. Instead, determine which value is correct or define a consistent rule such as:

• keep the most recent record
• keep the record from the trusted source
• merge complementary fields
• send the conflict for manual review

Standardize before detecting near-duplicates

Near-duplicates may differ because of formatting rather than meaning.

Useful preparation can include:

• trimming extra spaces
• standardizing capitalization
• normalizing dates and phone numbers
• correcting known category variations

This can reveal records that represent the same entity but were written differently.

Handle duplicates before data splitting

When possible, identify duplicate or related records before creating training and evaluation splits.

If copies of the same observation appear in both training and test data, the model may be evaluated on information it effectively already saw during training.

This can inflate test performance.

Keep a record of what was removed

Document the duplicate rule and the number of records affected.

This makes preprocessing reproducible and helps detect upstream data-quality problems.

In short, handle duplicate records by first confirming whether they are truly redundant, removing confirmed accidental copies, resolving conflicting records with clear rules, and preventing duplicates from leaking across training and evaluation data.