What is an encoder-decoder language model?

An encoder-decoder language model is a Transformer-based model that uses two separate components: an encoder to understand the input and a decoder to generate the output.

It is especially useful when one sequence must be transformed into another.

How does it work?

Suppose the input is:

Translate to French: Good morning

The model works in two stages.

First, the encoder processes the complete input and creates contextual representations of it.

Then, the decoder uses those encoder representations while generating the output token by token:

Input text
   ↓
Encoder
   ↓
Contextual representation
   ↓
Decoder
   ↓
Generated output

What does the encoder do?

The encoder usually uses bidirectional self-attention, allowing each input token to use information from tokens on both sides.

Its job is to build useful representations of the input sequence.

For example, in translation, it helps represent the meaning and relationships in the source sentence before generation begins.

What does the decoder do?

The decoder generates the output autoregressively.

It typically uses:

• Causal self-attention to look at previously generated output tokens
• Cross-attention to use information produced by the encoder

Conceptually:

previous output tokens
        ↓
causal self-attention
        ↓
cross-attention to encoder output
        ↓
next-token prediction

This lets the decoder generate a response while continuously referring back to the encoded input.

What tasks are encoder-decoder models useful for?

They are well suited to sequence-to-sequence tasks such as:

• Translation
• Summarization
• Text rewriting
• Question answering
• Structured text generation
• Converting one text format into another

The architecture is useful when there is a clear input sequence and a separate output sequence.

How is this different from a decoder-only LLM?

A decoder-only model processes the prompt and generated continuation through one causal Transformer stack.

An encoder-decoder model separates those roles:

Encoder → understand the input
Decoder → generate the output

The decoder can directly attend to the encoder's representations through cross-attention.

Examples

Well-known encoder-decoder Transformer families include T5 and BART.

Their exact training methods differ, but both use the general encoder-decoder structure.

For the other major architecture types, see “What is an encoder-only language model?” and “What is a decoder-only LLM?”

In short: an encoder-decoder language model first encodes the input into contextual representations and then uses a separate decoder to generate the output token by token, making the architecture particularly suitable for sequence-to-sequence tasks.