What is the difference between causal and masked language models?

The main difference between causal and masked language models is the context they are allowed to use when predicting tokens.

A causal language model predicts the next token using only the tokens that came before it.

A masked language model predicts hidden tokens using surrounding context, often from both the left and the right.

Simple example

Consider this sentence:

The cat is sleeping on the sofa.

A causal language model might learn:

The → predict cat
The cat → predict is
The cat is → predict sleeping

When predicting sleeping, it cannot look ahead at on the sofa.

A masked language model might instead see:

The cat is [MASK] on the sofa.

It can use both:

The cat is

and:

on the sofa

to predict sleeping.

Key differences

| Aspect | Causal language model | Masked language model | |---|---|---| | Prediction objective | Predict the next token | Predict hidden tokens | | Context used | Previous tokens only | Usually both left and right context | | Attention pattern | Causal / one-directional | Bidirectional | | Typical strength | Text generation | Text understanding | | Common example | GPT-style models | BERT-style models |

Why causal models are good at generation

During text generation, future tokens do not exist yet.

A causal language model is trained under that same restriction, so it naturally generates text one token at a time:

context → predict token → append token → repeat

This makes causal models especially suitable for chat, writing, code generation, and other generative tasks.

Why masked models are good at understanding

Masked language models can examine the full surrounding sentence when predicting a missing token.

That bidirectional context is useful for tasks such as:

• Text classification
• Named-entity recognition
• Sentiment analysis
• Search and retrieval
• Extractive question answering

Are they completely separate ideas?

They use different training objectives, but both learn statistical relationships between tokens and can be built with Transformer architectures.

For the individual concepts, see “What is a causal language model?” and “What is a masked language model?”

In short: causal language models predict future tokens from past context and are naturally suited to generation, while masked language models predict hidden tokens using surrounding context and are especially useful for understanding-oriented tasks.