What is autoregressive language modeling?

Autoregressive language modeling is a way of modeling text where the model predicts each token using only the tokens that came before it.

In other words, the model generates a sequence from left to right:

token 1 → token 2 → token 3 → token 4 → ...

At each step, the model estimates the probability of the next token given the previous tokens.

Mathematically, a sequence can be represented as:

P(x₁, x₂, ..., xₙ)
= P(x₁) × P(x₂ | x₁) × P(x₃ | x₁, x₂) × ... × P(xₙ | x₁, ..., xₙ₋₁)

Here, each token depends on the tokens that appear before it.

Example

Suppose the sequence begins with:

Artificial intelligence can

The model predicts probabilities for the next token, such as:

help      0.31
improve   0.18
generate  0.12
...

If help is selected, the context becomes:

Artificial intelligence can help

The model then predicts the next token again.

Why is it called autoregressive?

The term autoregressive comes from the idea that each new prediction depends on earlier values in the same sequence.

For language models, that means previously observed or generated tokens become the input used to predict the next token.

How is it used in LLMs?

Many generative LLMs use autoregressive modeling during both training and generation.

During training, the model learns to predict the next token from previous tokens.

During generation, it repeatedly:

predict next token
      ↓
append token
      ↓
predict again
      ↓
repeat

This is closely related to next-token prediction. See “What is next-token prediction in LLMs?” for the prediction objective itself.

Important limitation

Because generation happens token by token, later outputs depend on earlier generated tokens. An incorrect or poor token choice can influence the rest of the generated sequence.

In short: autoregressive language modeling predicts a text sequence one token at a time, with each prediction conditioned on the tokens that came before it.