How are Large Language Models pretrained?

Large Language Models are pretrained by exposing them to very large datasets and repeatedly training them to solve a language-modeling objective.

For many modern generative LLMs, that objective is next-token prediction.

1. Build a large training corpus

Pretraining begins with a large collection of data.

Depending on the model, the corpus may include:

• Web pages
• Books
• Articles
• Reference material
• Code
• Forums or discussions
• Other text sources

The exact mixture varies by model.

2. Clean and prepare the data

Raw data usually needs preprocessing before training.

Typical steps can include:

• Removing low-quality or malformed content
• Deduplicating repeated documents
• Filtering unwanted material
• Normalizing text formats
• Splitting data into training examples

These steps help improve the quality of the training signal.

3. Tokenize the text

The text is converted into tokens using the model's tokenizer.

Conceptually:

raw text
   ↓
tokenizer
   ↓
token IDs

Those token IDs become the input sequences used during training.

4. Create the training objective

For a decoder-only LLM, the model commonly learns to predict the next token.

For example:

Input:
Machine learning is a branch of

Target:
artificial

The same sequence provides many training targets because each position can be trained to predict the following token.

5. Run the forward pass

The token sequence passes through the Transformer.

The model produces probability distributions over the vocabulary for the tokens it must predict.

Its predictions are then compared with the correct target tokens.

6. Calculate the loss

A loss function measures how different the model's predictions are from the correct targets.

If the correct token receives low probability, the loss is higher.

If the model assigns higher probability to the correct token, the loss becomes lower.

7. Update the model parameters

Using backpropagation and an optimization algorithm, the training system calculates how the model's parameters should change to reduce the loss.

Conceptually:

prediction
    ↓
compare with target
    ↓
calculate loss
    ↓
backpropagation
    ↓
update parameters

This process is repeated across enormous numbers of token sequences.

8. Repeat across the dataset

Training continues over many batches of data.

Over time, the model gradually learns statistical patterns related to:

• Language structure
• Meaning and associations
• Facts present in the data
• Writing styles
• Code patterns
• Common forms of reasoning found in the corpus

What happens after pretraining?

The result is usually a base model.

It can then go through additional stages such as:

• Supervised fine-tuning
• Instruction tuning
• Preference optimization
• Other post-training methods

These later stages adapt the pretrained model toward specific behaviors.

In short: Large Language Models are pretrained by tokenizing enormous datasets, repeatedly predicting training targets such as the next token, measuring prediction error, and updating billions of model parameters through backpropagation until the model learns broad language patterns.