What is ALiBi in Large Language Models?

ALiBi stands for Attention with Linear Biases. It is a positional method used in Transformer models to give attention information about how far apart tokens are in a sequence.

Instead of adding positional embeddings to token representations, ALiBi modifies the attention scores directly.

What is the main idea?

In ALiBi, attention scores receive a distance-based bias.

Tokens that are farther apart receive a larger penalty.

Conceptually:

attention score
      +
distance-based bias
      ↓
adjusted attention score

This gives the model information about relative token distance without adding a separate positional embedding.

How does the bias work?

Suppose one token is attending to earlier tokens.

The farther an earlier token is from the current position, the more its attention score is reduced.

Conceptually:

nearby token   → small penalty
farther token  → larger penalty
very far token → even larger penalty

The penalty grows approximately linearly with distance.

Does every attention head use the same bias?

No.

Different attention heads use different slopes.

One head may penalize distant tokens strongly, while another may penalize them more gradually.

This allows different heads to operate over different effective distance ranges.

How is ALiBi different from positional embeddings?

Traditional positional embeddings add position information to token representations.

ALiBi instead changes the attention score itself:

Positional embeddings
→ modify token representations

ALiBi
→ modifies attention scores

This keeps the positional mechanism relatively simple.

How is ALiBi different from RoPE?

Both give Transformers positional information, but they do it differently.

• RoPE rotates query and key representations according to position.
• ALiBi adds a distance-dependent bias to attention scores.

So RoPE changes the representations used to calculate attention, while ALiBi changes the resulting attention scores.

Why is ALiBi useful for long sequences?

Because ALiBi is based on relative distance rather than a fixed learned embedding for each absolute position, it can generalize more naturally to sequence lengths beyond those used during training.

However, this does not mean a model can handle arbitrarily long context without quality or compute limitations.

Does ALiBi replace self-attention?

No.

ALiBi is an addition to the attention calculation.

The model still performs self-attention; ALiBi simply introduces a positional bias into the attention scores.

In short: ALiBi provides positional information by adding head-specific linear distance penalties to attention scores. Nearby tokens receive smaller penalties, distant tokens receive larger ones, and the method avoids using separate positional embeddings.