What is a pretraining objective in an LLM?

A pretraining objective is the task an LLM is trained to solve repeatedly during its initial training stage.

The objective defines what the model is asked to predict and therefore determines the learning signal used to update its parameters.

What does a pretraining objective do?

A pretraining objective turns raw text into a learning problem.

For example, a decoder-only LLM may receive:

The largest planet in the solar system is

and learn to predict the next token:

Jupiter

The model's prediction is compared with the correct target, a loss is calculated, and the parameters are updated.

What is the most common objective for generative LLMs?

Many decoder-only LLMs use causal language modeling, also called next-token prediction.

The model learns:

previous tokens
      ↓
predict the next token

Because each position in a sequence can provide a prediction target, enormous amounts of unlabeled text can be used for training.

Are there other pretraining objectives?

Yes.

Different language-model architectures can use different objectives.

Common examples include:

• Causal language modeling — predict the next token using earlier tokens.
• Masked language modeling — hide selected tokens and predict them from surrounding context.
• Denoising objectives — corrupt part of an input sequence and train the model to reconstruct the original content.

The exact objective depends on the architecture and intended behavior of the model.

Why does the objective matter?

The objective influences what kind of representations and capabilities the model learns naturally.

For example:

• Causal language modeling naturally supports autoregressive text generation.
• Masked language modeling encourages bidirectional understanding of context.
• Denoising objectives are useful for models that transform one sequence into another.

So the objective is closely connected to the model architecture.

Does the objective require manually labeled data?

Usually not during large-scale pretraining.

Most LLM pretraining is self-supervised because the targets are generated from the data itself.

For example, in next-token prediction:

input tokens → earlier part of the sequence
target token → actual next token

No human needs to manually label every example.

Is the pretraining objective the same as the final task?

No.

A model may be pretrained only to predict tokens, yet later perform tasks such as:

• Question answering
• Summarization
• Translation
• Coding
• Conversation

Those capabilities emerge from what the model learns during pretraining and can be further shaped through fine-tuning and post-training.

For the two common objectives, see “What is causal language modeling during pretraining?” and “What is masked language modeling during pretraining?”

In short: a pretraining objective defines the prediction task an LLM repeatedly solves during initial training. It creates the learning signal that drives parameter updates and strongly influences the capabilities the model develops.