What is a causal language model?

A causal language model is a language model that predicts each token using only the tokens that come before it in the sequence.

It is called causal because information is allowed to flow in one direction: from earlier tokens to later tokens.

For example, given:

The capital of Italy is

the model can use all of those earlier tokens to predict the next token, such as:

Rome

How does a causal language model enforce this?

In Transformer-based causal language models, self-attention uses a causal mask.

The mask prevents a token from looking at future tokens during training.

For a sequence such as:

I love machine learning

the model can use:

I → predict love
I love → predict machine
I love machine → predict learning

But when predicting love, it is not allowed to see machine learning.

Why is this useful?

This training setup matches how text is generated at inference time.

During generation, future tokens do not exist yet, so the model must predict the next token using only the current context.

That makes causal language models well suited for:

• Text generation
• Conversation
• Code generation
• Story completion
• Autocomplete
• Instruction-following systems

Is a causal language model autoregressive?

Usually, yes.

A causal language model generates text autoregressively by repeatedly predicting the next token from previous tokens.

The key distinction is:

• Causal describes the attention constraint: the model cannot use future tokens.
• Autoregressive describes the sequence modeling process: each prediction depends on earlier tokens.

For the broader sequence-generation idea, see “What is autoregressive language modeling?”

Causal vs. masked language modeling

A causal language model predicts tokens from left-to-right context.

A masked language model can use tokens on both sides of a masked position.

That distinction is covered separately in “What is a masked language model?”

In short: a causal language model predicts each token using only earlier tokens, typically through a causal attention mask that blocks access to future positions.