What is the difference between encoder-only and decoder-only language models?

Encoder-only and decoder-only language models are both based on the Transformer architecture, but they are designed for different kinds of language tasks.

The main difference is how they process context and what they are optimized to produce.

Encoder-only models

Encoder-only models process the input sequence as a whole.

They typically use bidirectional self-attention, which allows each token to use information from tokens that appear both before and after it.

This makes them well suited to understanding and representing text.

Common tasks include:

• Text classification
• Sentiment analysis
• Named-entity recognition
• Semantic search
• Embedding generation
• Extractive question answering

A well-known example is BERT.

Decoder-only models

Decoder-only models are designed primarily for autoregressive generation.

They use causal self-attention, which prevents a token from seeing future tokens while predicting the next token.

Conceptually:

Encoder-only:
past ← token → future

Decoder-only:
past → next-token prediction

Decoder-only models are commonly used for:

• Chat
• Text generation
• Code generation
• Summarization
• Question answering
• Instruction following

How do their training objectives differ?

Encoder-only models are commonly associated with objectives such as masked language modeling, where selected tokens are hidden and predicted using surrounding context.

Decoder-only models are commonly trained with causal language modeling, where the model predicts the next token from the tokens that came before it.

These training objectives influence the kinds of tasks each architecture handles naturally.

How do their outputs differ?

An encoder-only model normally produces contextual representations of the input tokens.

Those representations can then be used by another layer for tasks such as classification or retrieval.

A decoder-only model produces probabilities for the next token and repeatedly generates new tokens.

So:

Encoder-only
→ represent and understand input

Decoder-only
→ generate continuation

Does this mean decoder-only models cannot understand text?

No.

Decoder-only LLMs can perform many language-understanding tasks because the prompt, instructions, and relevant context are all provided before the answer is generated.

The distinction is architectural and training-related, not a statement that one architecture can only “understand” and the other can only “generate.”

Which architecture is better?

Neither is universally better.

The appropriate architecture depends on the task.

Encoder-only models can be efficient for representation and classification tasks, while decoder-only models are especially flexible for open-ended generation and instruction-following systems.

For the individual architectures, see “What is an encoder-only language model?” and “What is a decoder-only LLM?”

In short: encoder-only models use bidirectional attention to build rich representations of the full input, while decoder-only models use causal attention to predict and generate tokens sequentially.