What is multi-head attention in an LLM?

Multi-head attention is an attention mechanism where an LLM performs several attention calculations in parallel instead of relying on a single attention pattern.

Each attention calculation is called an attention head.

Why use multiple heads?

Different heads can learn to focus on different kinds of relationships in the same sequence.

For example, one head may focus more on:

• Nearby grammatical relationships
• Long-distance dependencies
• References between words
• Important instruction tokens
• Other useful token interactions

The model does not manually assign these roles. They are learned during training.

How does it work?

The token representations are projected into separate query, key, and value representations for each head.

Conceptually:

Token representations
        ↓
 ┌──────┼──────┐
 ↓      ↓      ↓
Head 1 Head 2 Head 3 ...
 ↓      ↓      ↓
attention outputs
 └──────┼──────┘
        ↓
combine heads
        ↓
final attention output

Each head computes its own attention weights and produces its own output.

The outputs from all heads are then combined and transformed before being passed to the next part of the Transformer layer.

What does each head see?

In standard multi-head attention, every head typically receives information derived from the same token sequence, but each head uses its own learned projections.

That means two heads can assign very different attention weights to the same tokens.

Does more attention heads always mean a better model?

No.

The number of heads is an architectural choice.

Using more heads can increase the number of distinct attention patterns the model can represent, but model quality also depends on factors such as:

• Hidden dimension
• Number of layers
• Training data
• Training objective
• Overall architecture
• Compute budget

More heads alone do not guarantee better performance.

How is this related to query, key, and value?

Each attention head works with query, key, and value representations.

In standard multi-head attention, different heads usually have separate learned Q, K, and V projections.

For the underlying representations, see “What are query, key, and value representations in LLM attention?”

Are there alternatives?

Yes.

Modern LLMs may use alternatives such as:

• Multi-query attention (MQA)
• Grouped-query attention (GQA)

These approaches reduce some of the memory and computation associated with standard multi-head attention, especially during inference.

They are covered separately in their dedicated articles.

In short: multi-head attention lets an LLM compute several attention patterns in parallel, allowing different heads to capture different relationships between tokens before their outputs are combined.