What is data deduplication in LLM pretraining?

Data deduplication in LLM pretraining is the process of finding and removing repeated or highly similar content from the training corpus.

The goal is to prevent the model from seeing the same information far more often than intended.

Why do duplicates appear?

Large training datasets are often assembled from many sources.

The same content can appear repeatedly because of:

• Mirrored websites
• Reposted articles
• Copied documentation
• Boilerplate pages
• Dataset overlap
• Multiple versions of the same document
• Quoted or duplicated passages

Without deduplication, these repeated examples can become overrepresented.

What kinds of duplicates are removed?

Deduplication can happen at different levels.

Common examples include:

• Exact document duplicates — two documents are identical.
• Near-duplicate documents — documents differ only slightly.
• Repeated passages — large sections of text appear across multiple documents.
• Repeated code — identical or very similar code blocks appear in many places.

A training pipeline may use one or several of these approaches.

How are exact duplicates detected?

Exact duplicates are relatively straightforward.

A system can normalize a document and compute a fingerprint or hash.

Conceptually:

document
   ↓
normalize
   ↓
fingerprint
   ↓
compare fingerprints
   ↓
remove repeated copies

Documents with the same fingerprint can be treated as duplicates.

How are near-duplicates detected?

Near-duplicate detection is harder because two documents may contain nearly the same content with small changes.

For example:

Document A:
LLMs learn from large datasets.

Document B:
Large language models learn from very large datasets.

Methods can compare document shingles, hashes, similarity signatures, or other representations to estimate whether two pieces of content are substantially the same.

Why is deduplication useful?

It can help:

• Reduce unnecessary training computation
• Prevent repeated content from dominating the dataset
• Reduce memorization pressure
• Improve dataset diversity
• Reduce overlap between training and evaluation material when applied carefully

It also makes the training corpus more efficient because fewer tokens are spent relearning identical material.

Is deduplication always simple?

No.

Aggressive deduplication can accidentally remove useful examples that happen to be similar.

For example, repeated technical definitions, common code patterns, or legitimate translations may look highly similar even though keeping some of them is useful.

So the similarity threshold and deduplication method matter.

Is deduplication the same as data cleaning?

Deduplication is one part of the broader data-cleaning process.

Conceptually:

Data cleaning
├── noise removal
├── quality filtering
├── language filtering
├── deduplication
└── other preprocessing

For the broader process, see “How is training data cleaned for LLMs?”

In short: data deduplication removes exact or near-repeated content from an LLM's pretraining corpus so repeated examples do not consume unnecessary compute or become disproportionately influential during training.