How is training data cleaned for LLMs?

Training data is cleaned for LLMs by removing or transforming low-quality, duplicated, malformed, irrelevant, or otherwise unsuitable content before it is used for pretraining.

The goal is not to make every document perfect. It is to improve the overall quality of the training signal.

1. Remove obvious noise

Raw data often contains material that is not useful language content, such as:

• Broken markup
• Navigation menus
• Repeated headers and footers
• Tracking text
• Corrupted characters
• Empty or nearly empty pages
• Machine-generated boilerplate

Cleaning removes or reduces this noise.

2. Normalize the text

Different sources may use inconsistent formatting or encodings.

A preprocessing pipeline may normalize things such as:

• Character encoding
• Whitespace
• Line breaks
• Unicode characters
• Document structure

This creates more consistent input for later stages.

3. Filter very low-quality content

Datasets may contain spam, keyword stuffing, gibberish, automatically generated pages, or other poor-quality text.

Quality filters can use rules, statistical signals, classifiers, or combinations of methods to remove such material.

4. Detect language

For multilingual datasets, language-detection systems can estimate the language of each document.

This helps the training pipeline:

• Remove incorrectly identified content
• Control the amount of data per language
• Build a desired multilingual mixture

5. Remove duplicates

The same document or passage can appear many times across different sources.

Deduplication helps prevent the model from seeing identical or nearly identical content too frequently.

Because this is an important topic by itself, it is covered separately in “What is data deduplication in LLM pretraining?”

6. Filter sensitive or unsuitable data

Depending on the training pipeline, additional filters may be used to identify material such as:

• Personally identifiable information
• Secrets or credentials
• Certain unsafe or restricted content
• Data that should not be included because of policy or licensing constraints

The exact rules differ across datasets and organizations.

7. Score and select documents

Not all usable documents have the same quality.

A pipeline may assign quality scores based on signals such as:

• Readability
• Coherence
• Information density
• Source characteristics
• Repetition
• Formatting quality

Higher-quality material may be retained or given greater weight.

8. Prepare documents for tokenization

After filtering and cleaning, documents are converted into a consistent form suitable for tokenization and sequence construction.

Conceptually:

raw data
   ↓
noise removal
   ↓
normalization
   ↓
quality filtering
   ↓
deduplication
   ↓
final corpus
   ↓
tokenization

Can aggressive cleaning cause problems?

Yes.

If filtering is too strict, useful content, minority languages, informal writing, or unusual domains may be removed.

So data cleaning involves a trade-off between removing noise and preserving useful diversity.

In short: LLM training data is cleaned through normalization, noise removal, quality filtering, language detection, deduplication, and other selection rules. The aim is to create a high-quality and diverse corpus before tokenization and pretraining begin.