How do you detect duplicate records in a dataset?

Duplicate records are detected by checking whether two or more rows represent the same observation, entity, or event.

Some duplicates are exact copies, while others differ slightly because of formatting, timestamps, spelling, or data-entry differences.

Start with exact duplicates

An exact duplicate occurs when all relevant fields are identical.

For example:

Customer ID: 1024
Age: 35
City: Delhi
Plan: Premium

If the same row appears more than once, it may be an exact duplicate.

Most data-processing tools can identify rows that repeat across all columns.

Check duplicates using unique identifiers

If the dataset contains a reliable identifier such as:

• customer ID
• transaction ID
• order ID
• device ID

check whether the same identifier appears more times than expected.

A repeated identifier is not always an error. For example, one customer may legitimately have many transactions.

The meaning of the identifier must therefore be understood before declaring records duplicates.

Check combinations of columns

Sometimes no single column uniquely identifies a record.

In that case, use a combination such as:

customer_id + transaction_date + amount

Repeated combinations can reveal duplicate events.

Look for near-duplicates

Near-duplicates represent the same real-world record but are not exactly identical.

Examples include:

• Rahul Sharma vs Rahul Sharma
• Bengaluru vs Bangalore
• different capitalization
• small spelling differences
• slightly different timestamps for the same event

These may require normalization, fuzzy matching, or domain-specific rules.

Check duplicates across data splits

Duplicate or nearly identical records appearing in both training and test data can make model evaluation overly optimistic.

For grouped data, also check whether multiple records from the same entity should remain in the same split.

Always investigate before removing

A repeated row is not automatically incorrect.

Some datasets legitimately contain repeated events or measurements, so duplicate detection should be followed by a check of what each row represents.

In short, detect duplicate records by checking exact row matches, repeated identifiers, repeated combinations of key fields, and near-duplicate patterns, while confirming that the repeated records are not legitimate observations.