Large Language Models

Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.

How is LLM training data collected?

LLM training data is collected by assembling large amounts of text and other relevant content from many different sources, then preparing that material for later filtering, cleaning, deduplication, and tokenization. The exact collection process differs across organizations and models. Where can the data come from? Common sources include: Publicly available web pages Books and publications Open datasets Technical and reference documents Source code repositories Licensed datasets Data supplied through partnerships Organization owned or proprietary data Human created training examples Synthetic data generated by models or other systems A general purpose LLM usually combines many of these sources rather than relying on one dataset. How is public web data collected? Large web datasets can be created by crawling publicly accessible pages or by using existing web archives and datasets. Conceptually: The raw collection can contain duplicates, spam, navigation text, malformed pages, and low quality material, so collection is only the beginning of the pipeline. How is licensed data collected? Some training data is obtained through licenses or agreements with publishers, data providers, platforms, or other rights holders. Instead of crawling the material, the model developer receives access to an approved dataset or content collection under the relevant agreement. Can organizations use their own data? Yes. A model designed for a particular organization or domain may include internally owned data such as: Documentation Knowledge bases Manuals Support material Domain specific text Whether that data is appropriate for training depends on permissions, privacy requirements, and the intended use of the model. Is human created data collected too? Yes. Human annotators or subject matter experts can create examples such as: Questions and answers Instructions and responses Demonstrations Preference comparisons However, these kinds of datasets are often more important during fine tuning and post training than during the initial large scale pretraining stage. What about synthetic data? Synthetic examples can be generated automatically to expand or target particular kinds of training data. For example, a system might generate: Synthetic data still requires quality control because generated mistakes can become part of the training signal. Is collected data immediately used for training? Usually not. After collection, it normally passes through additional stages such as: Those later preparation stages help turn raw collected material into a usable pretraining corpus. Are permissions and privacy relevant? Yes. Training data collection must consider factors such as: Data access rights Licensing terms Privacy Sensitive information Dataset provenance Applicable policies and legal requirements The specific requirements depend on the data source and jurisdiction. In short: LLM training data is collected from a mixture of public, licensed, proprietary, human created, and synthetic sources. The raw material is then passed through a preparation pipeline before it becomes suitable for model training.

How is training data cleaned for LLMs?

Training data is cleaned for LLMs by removing or transforming low quality, duplicated, malformed, irrelevant, or otherwise unsuitable content before it is used for pretraining. The goal is not to make every document perfect. It is to improve the overall quality of the training signal. 1. Remove obvious noise Raw data often contains material that is not useful language content, such as: Broken markup Navigation menus Repeated headers and footers Tracking text Corrupted characters Empty or nearly empty pages Machine generated boilerplate Cleaning removes or reduces this noise. 2. Normalize the text Different sources may use inconsistent formatting or encodings. A preprocessing pipeline may normalize things such as: Character encoding Whitespace Line breaks Unicode characters Document structure This creates more consistent input for later stages. 3. Filter very low quality content Datasets may contain spam, keyword stuffing, gibberish, automatically generated pages, or other poor quality text. Quality filters can use rules, statistical signals, classifiers, or combinations of methods to remove such material. 4. Detect language For multilingual datasets, language detection systems can estimate the language of each document. This helps the training pipeline: Remove incorrectly identified content Control the amount of data per language Build a desired multilingual mixture 5. Remove duplicates The same document or passage can appear many times across different sources. Deduplication helps prevent the model from seeing identical or nearly identical content too frequently. Because this is an important topic by itself, it is covered separately in “What is data deduplication in LLM pretraining?” 6. Filter sensitive or unsuitable data Depending on the training pipeline, additional filters may be used to identify material such as: Personally identifiable information Secrets or credentials Certain unsafe or restricted content Data that should not be included because of policy or licensing constraints The exact rules differ across datasets and organizations. 7. Score and select documents Not all usable documents have the same quality. A pipeline may assign quality scores based on signals such as: Readability Coherence Information density Source characteristics Repetition Formatting quality Higher quality material may be retained or given greater weight. 8. Prepare documents for tokenization After filtering and cleaning, documents are converted into a consistent form suitable for tokenization and sequence construction. Conceptually: Can aggressive cleaning cause problems? Yes. If filtering is too strict, useful content, minority languages, informal writing, or unusual domains may be removed. So data cleaning involves a trade off between removing noise and preserving useful diversity. In short: LLM training data is cleaned through normalization, noise removal, quality filtering, language detection, deduplication, and other selection rules. The aim is to create a high quality and diverse corpus before tokenization and pretraining begin.

What is data deduplication in LLM pretraining?

Data deduplication in LLM pretraining is the process of finding and removing repeated or highly similar content from the training corpus. The goal is to prevent the model from seeing the same information far more often than intended. Why do duplicates appear? Large training datasets are often assembled from many sources. The same content can appear repeatedly because of: Mirrored websites Reposted articles Copied documentation Boilerplate pages Dataset overlap Multiple versions of the same document Quoted or duplicated passages Without deduplication, these repeated examples can become overrepresented. What kinds of duplicates are removed? Deduplication can happen at different levels. Common examples include: Exact document duplicates — two documents are identical. Near duplicate documents — documents differ only slightly. Repeated passages — large sections of text appear across multiple documents. Repeated code — identical or very similar code blocks appear in many places. A training pipeline may use one or several of these approaches. How are exact duplicates detected? Exact duplicates are relatively straightforward. A system can normalize a document and compute a fingerprint or hash. Conceptually: Documents with the same fingerprint can be treated as duplicates. How are near duplicates detected? Near duplicate detection is harder because two documents may contain nearly the same content with small changes. For example: Methods can compare document shingles, hashes, similarity signatures, or other representations to estimate whether two pieces of content are substantially the same. Why is deduplication useful? It can help: Reduce unnecessary training computation Prevent repeated content from dominating the dataset Reduce memorization pressure Improve dataset diversity Reduce overlap between training and evaluation material when applied carefully It also makes the training corpus more efficient because fewer tokens are spent relearning identical material. Is deduplication always simple? No. Aggressive deduplication can accidentally remove useful examples that happen to be similar. For example, repeated technical definitions, common code patterns, or legitimate translations may look highly similar even though keeping some of them is useful. So the similarity threshold and deduplication method matter. Is deduplication the same as data cleaning? Deduplication is one part of the broader data cleaning process. Conceptually: For the broader process, see “How is training data cleaned for LLMs?” In short: data deduplication removes exact or near repeated content from an LLM's pretraining corpus so repeated examples do not consume unnecessary compute or become disproportionately influential during training.

Why is data deduplication important for LLMs?

Data deduplication is important for LLMs because repeated content can distort the training distribution, waste compute, and increase the chance that a model memorizes specific examples. The purpose of deduplication is not just to make the dataset smaller. It is to make the training signal more balanced and efficient. 1. It reduces wasted training compute If the same document appears many times, the model may repeatedly learn from nearly identical examples. Conceptually: Removing unnecessary duplicates allows more of the training budget to be spent on unique information. 2. It prevents repeated content from becoming overrepresented Suppose one article appears once and another appears hundreds of times because it was copied across many websites. Without deduplication, the repeated article can have a much larger influence on training than intended. Deduplication helps keep the dataset closer to the desired distribution. 3. It can reduce memorization pressure Seeing the same passage many times makes it easier for a model to memorize that specific content. Reducing repetition can lower this pressure, although deduplication alone does not eliminate memorization. 4. It improves dataset diversity A fixed training budget can contain either many repeated examples or a broader variety of unique examples. Removing duplicates makes room for more diverse material: This can improve the usefulness of the corpus. 5. It can make evaluation more reliable A serious problem occurs when benchmark or evaluation examples also appear in the training data. If the model has already seen the same or nearly identical questions during training, benchmark scores may overstate its true ability to generalize. Deduplication and contamination checks can help reduce this overlap. 6. Near duplicates matter too Removing only perfectly identical documents is not always enough. For example: These are not exact duplicates, but large datasets can contain many highly similar versions of the same content. Near duplicate detection helps reduce that repetition. Can deduplication remove too much? Yes. If similarity thresholds are too aggressive, useful examples may be removed. Common definitions, legitimate code patterns, translations, templates, or naturally repeated phrases can look similar without being unwanted duplicates. So deduplication requires a balance between: Removing unnecessary repetition Preserving meaningful diversity Is deduplication the same as benchmark decontamination? No. General deduplication removes repeated material within the training corpus. Benchmark decontamination specifically looks for overlap between training data and evaluation datasets. The two are related, but they solve different problems. For the core process, see “What is data deduplication in LLM pretraining?” In short: data deduplication matters because it reduces wasted compute, limits accidental overrepresentation and memorization, improves dataset diversity, and can help make model evaluation more trustworthy.

What is data contamination in LLM training?

Data contamination in LLM training happens when information that should remain outside the training set accidentally appears in the model's training data. The most important case is when evaluation or benchmark examples are included in the training corpus. Why is contamination a problem? Suppose a benchmark is meant to test whether a model can solve a problem it has never seen before. If the exact question, answer, or a very close version already appeared during training, the model may perform well because it has effectively seen the test before. Conceptually: What can become contaminated? Contamination can involve: Benchmark questions Benchmark answers Test set examples Evaluation prompts Reference solutions Near duplicate versions of evaluation data Even partial overlap can matter if it gives the model information that should have remained unseen. How does contamination happen? Large pretraining corpora are collected from many sources. A benchmark may already be publicly available on: Websites Repositories Papers Forums Tutorials Dataset mirrors If those sources are included in the training corpus, benchmark content can enter the training data unintentionally. Is exact duplication the only problem? No. Near duplicates can also create contamination. For example, a benchmark problem may appear online with slightly different wording but the same solution. So contamination checks often need to look beyond exact text matching. Why does contamination affect evaluation? A benchmark is useful when it measures how well a model generalizes to unseen examples. Contamination weakens that assumption. A high score may then reflect: Memorization Familiarity with the benchmark Exposure to similar answers rather than genuine generalization. How can contamination be reduced? Possible approaches include: Comparing training data with benchmark datasets Removing exact and near duplicate matches Keeping evaluation sets private when possible Tracking dataset provenance Using newly created or refreshed benchmarks Testing models on data created after training No method is perfect, especially when the full training corpus is extremely large. Is data contamination the same as general data duplication? No. Deduplication removes repeated material within the training corpus. Contamination detection focuses on preventing inappropriate overlap between training data and data intended for evaluation or another held out purpose. For the evaluation specific case, see “What is benchmark contamination in LLMs?” In short: data contamination occurs when information that should be unseen—especially evaluation data—appears in an LLM's training corpus. This can make model performance look better than its true ability to generalize.

What is benchmark contamination in LLMs?

Benchmark contamination in LLMs happens when examples from an evaluation benchmark, or very similar versions of them, appear in the model's training data. This creates a form of train test leakage and can make benchmark scores look better than the model's true ability to generalize. Why is benchmark contamination a problem? A benchmark is supposed to test the model on examples it has not already learned directly from. If benchmark questions or answers were present during training, the model may perform well because it has seen them before. Conceptually: What kinds of overlap can cause contamination? Contamination can involve: Exact benchmark questions Exact reference answers Near duplicate questions Reworded versions of the same problem Solutions copied into tutorials or repositories Benchmark examples reproduced in papers, blogs, or forums So checking only for exact duplicates may not be enough. How does benchmark data enter training corpora? Many benchmarks are publicly available. Their examples may be copied into: Websites Git repositories Research papers Tutorials Discussion forums Dataset mirrors If these sources are later included in a large pretraining corpus, benchmark examples can enter the training data unintentionally. How can contamination inflate scores? Suppose a model is evaluated on a reasoning problem that already appeared in its training corpus. A correct answer might reflect: Memorization Familiarity with the specific problem Exposure to the reference solution rather than the ability to solve a genuinely unseen problem. This makes it harder to compare models fairly. How can benchmark contamination be detected? Common approaches include: Exact string matching Hash based matching N gram overlap checks Near duplicate detection Similarity based matching Searching for benchmark questions and answers in known training sources For very large or partially undisclosed training corpora, perfect detection can be difficult. How can contamination be reduced? Possible strategies include: Removing benchmark matches from training data Keeping some evaluation sets private Creating fresh benchmark examples Using evaluation data produced after the model's training cutoff Tracking dataset provenance Running decontamination checks before training or evaluation Is benchmark contamination the same as overfitting? No. Overfitting usually means a model fits its training data too closely and generalizes poorly. Benchmark contamination means evaluation material itself leaked into training. A model can be contaminated even if it is not generally overfitted. Is benchmark contamination the same as general data contamination? Benchmark contamination is a specific form of data contamination focused on evaluation datasets. General data contamination can refer more broadly to any information entering a training set when it should have remained separate. For the broader concept, see “What is data contamination in LLM training?” In short: benchmark contamination occurs when benchmark examples or close variants appear in an LLM's training data. This can inflate evaluation scores and make it harder to measure the model's true ability to generalize to unseen problems.

How does training data quality affect an LLM?

Training data quality has a major effect on an LLM because the model learns patterns from the data it is exposed to. If the training corpus contains useful, accurate, diverse, and well structured examples, the model receives a stronger learning signal. If the corpus contains large amounts of noise, duplication, misinformation, or low quality text, those patterns can also influence the model. What does “high quality data” mean? There is no single definition, but useful training data often has qualities such as: Clear and coherent language Relevant information Accurate or trustworthy content Good formatting Low duplication Meaningful diversity Appropriate coverage of languages and domains Quality also depends on the model's intended use. How can low quality data hurt an LLM? Poor quality training data can introduce problems such as: Learning incorrect associations Producing incoherent or low value text Overrepresenting spam or repetitive patterns Reinforcing biases present in the data Wasting training compute on unhelpful examples Reducing performance in underrepresented domains or languages For example, if a dataset contains many duplicated low quality pages, the model may spend significant training capacity learning those repeated patterns. Does factual accuracy matter? Yes. If incorrect information appears frequently in the training corpus, the model can learn statistical associations around that information. However, even a high quality corpus cannot guarantee that an LLM will always produce factual answers, because generation is still based on learned probability patterns rather than direct fact retrieval. Why does diversity matter alongside quality? A dataset can be individually high quality but still be too narrow. For example, if almost all training data comes from one writing style, language, region, or technical domain, the model may perform poorly outside that distribution. A strong training mixture therefore aims for both: Can too much filtering reduce quality? Yes. Aggressive filtering can remove noise, but it can also remove useful informal language, minority language content, unusual writing styles, or specialized material. So data curation involves a trade off between cleanliness and diversity. Is more high quality data always better? Generally, more useful data can improve training, but the value of additional data depends on: Model capacity Compute budget Data redundancy Domain coverage Training strategy At some point, repeatedly adding very similar examples gives smaller benefits than adding new, informative data. Does data quality matter more than model size? Both matter. A larger model trained on poor data can still learn undesirable or weak patterns, while a smaller model trained on carefully curated data may perform strongly on its target domain. Model size, data quantity, data quality, and compute need to be balanced. For related topics, see “How is training data cleaned for LLMs?” and “How does training data diversity affect an LLM?” In short: training data quality affects what an LLM learns, how efficiently it learns, and how reliably it performs. Cleaner, more accurate, diverse, and relevant data usually provides a better training signal than a larger but noisy or repetitive corpus.

How does training data diversity affect an LLM?

Training data diversity affects an LLM by determining how broadly the model is exposed to different languages, topics, writing styles, domains, and ways of expressing information. A diverse training corpus gives the model a wider range of patterns to learn from. What does data diversity mean? Diversity can refer to differences in: Topics Domains Languages Writing styles Document formats Geographic and cultural perspectives Formal and informal language Code and natural language Short and long documents A dataset can be very large but still be narrow if most of its content comes from similar sources. Why does diversity matter? If a model is trained on varied data, it has more opportunities to learn how language behaves across different situations. For example: This can improve the model's ability to handle different types of prompts. What happens when the data is too narrow? If the training corpus heavily favors one type of content, the model may perform better in that area and worse elsewhere. For example, a dataset dominated by technical English could lead to weaker performance on: Informal language Other languages Creative writing Specialized non technical domains Regional expressions The model reflects the distribution of the data it sees. Does diversity help multilingual performance? Yes. A multilingual model needs meaningful amounts of data from multiple languages. However, simply including a language is not enough. Performance also depends on: How much data exists for that language The quality of that data How balanced it is against other languages The kinds of tasks and domains represented A language with very little high quality data may still be underrepresented. Does diversity reduce bias? It can help reduce overrepresentation of a narrow viewpoint, but diversity alone does not guarantee an unbiased model. Bias can still come from: Imbalanced source selection Unequal representation Historical patterns in the data Filtering choices Annotation decisions So diversity is one part of responsible data curation, not a complete solution. Can too much diversity be a problem? Not by itself, but an unbalanced mixture can create trade offs. For example, adding large amounts of low quality or irrelevant data may dilute useful training signal. That is why training pipelines often balance: rather than maximizing only one of them. How does diversity affect specialization? A general purpose LLM benefits from broad diversity because it is expected to handle many topics and tasks. A specialized model may intentionally use a narrower corpus focused on one domain. So the right level of diversity depends on the model's intended use. For the related quality dimension, see “How does training data quality affect an LLM?” In short: training data diversity broadens the range of patterns an LLM can learn. A diverse corpus can improve coverage across languages, domains, styles, and tasks, while an overly narrow or imbalanced corpus can make model capability uneven.

What is a pretraining objective in an LLM?

A pretraining objective is the task an LLM is trained to solve repeatedly during its initial training stage. The objective defines what the model is asked to predict and therefore determines the learning signal used to update its parameters. What does a pretraining objective do? A pretraining objective turns raw text into a learning problem. For example, a decoder only LLM may receive: and learn to predict the next token: The model's prediction is compared with the correct target, a loss is calculated, and the parameters are updated. What is the most common objective for generative LLMs? Many decoder only LLMs use causal language modeling , also called next token prediction. The model learns: Because each position in a sequence can provide a prediction target, enormous amounts of unlabeled text can be used for training. Are there other pretraining objectives? Yes. Different language model architectures can use different objectives. Common examples include: Causal language modeling — predict the next token using earlier tokens. Masked language modeling — hide selected tokens and predict them from surrounding context. Denoising objectives — corrupt part of an input sequence and train the model to reconstruct the original content. The exact objective depends on the architecture and intended behavior of the model. Why does the objective matter? The objective influences what kind of representations and capabilities the model learns naturally. For example: Causal language modeling naturally supports autoregressive text generation. Masked language modeling encourages bidirectional understanding of context. Denoising objectives are useful for models that transform one sequence into another. So the objective is closely connected to the model architecture. Does the objective require manually labeled data? Usually not during large scale pretraining. Most LLM pretraining is self supervised because the targets are generated from the data itself. For example, in next token prediction: No human needs to manually label every example. Is the pretraining objective the same as the final task? No. A model may be pretrained only to predict tokens, yet later perform tasks such as: Question answering Summarization Translation Coding Conversation Those capabilities emerge from what the model learns during pretraining and can be further shaped through fine tuning and post training. For the two common objectives, see “What is causal language modeling during pretraining?” and “What is masked language modeling during pretraining?” In short: a pretraining objective defines the prediction task an LLM repeatedly solves during initial training. It creates the learning signal that drives parameter updates and strongly influences the capabilities the model develops.

What is causal language modeling during pretraining?

Causal language modeling is a pretraining objective where an LLM learns to predict the next token using only the tokens that come before it. It is the standard pretraining objective used by many decoder only language models. How does it work? Suppose the training sequence is: The model can learn several next token predictions from the same sequence: At each position, the model tries to predict the next token. Why is it called causal? The model is prevented from using future tokens when making a prediction. For example, when predicting: it can use: but it cannot look ahead at: This left to right restriction is enforced using causal self attention or a causal attention mask. What happens during training? A simplified training step looks like: This process is repeated across enormous amounts of tokenized data. Can multiple tokens in one sequence be training targets? Yes. A single sequence can provide a prediction target at many positions. For example: In practice, these token predictions can be computed efficiently across the sequence during training while still respecting the causal mask. Why is causal language modeling useful? It gives the model a simple self supervised objective. The training targets come directly from the original text, so no human needs to manually label each example. By predicting next tokens across massive datasets, the model can learn patterns involving: Grammar Semantics Facts and associations Writing styles Code Document structure How does this connect to text generation? The training objective closely matches inference. During generation: That is why causal language modeling is naturally suited to autoregressive text generation. How is it different from masked language modeling? Causal language modeling predicts the next token using only earlier context. Masked language modeling hides selected tokens and predicts them using surrounding context from both directions. That difference is covered separately in “What is masked language modeling during pretraining?” In short: causal language modeling trains an LLM to predict each next token from the tokens that came before it. This left to right objective is self supervised and naturally matches how decoder only LLMs generate text.