What is sparse attention in LLMs?

Sparse attention is an attention approach where each token attends to only a selected subset of other tokens instead of attending to every token in the sequence.

The goal is to reduce the computation and memory required by full self-attention, especially for long contexts.

Why is sparse attention needed?

In full self-attention, every token can interact with every other allowed token.

For a sequence with many tokens, the number of attention comparisons grows rapidly as the sequence becomes longer.

Sparse attention reduces those comparisons by keeping only selected connections.

Conceptually:

Full attention:
every token ↔ many or all other tokens

Sparse attention:
each token ↔ selected tokens only

How are the selected connections chosen?

Different sparse-attention designs use different patterns.

Common examples include:

• Local or sliding-window attention — tokens attend mainly to nearby tokens
• Global tokens — selected tokens can attend broadly across the sequence
• Strided or dilated patterns — attention skips across positions at regular intervals
• Block-sparse attention — attention is allowed only between selected blocks of tokens
• Hybrid patterns — combine local and long-range connections

The pattern is usually defined by the model architecture rather than chosen independently for every token at runtime.

How does sparse attention improve efficiency?

Because fewer query-key comparisons are calculated, sparse attention can reduce:

• Attention computation
• Memory usage
• Attention-score storage

The exact savings depend on the sparsity pattern.

For example, a fixed local window can make attention cost grow roughly with sequence length times the window size instead of with all token pairs.

What is the trade-off?

The main trade-off is reduced direct connectivity.

If two important tokens are far apart and the attention pattern does not connect them directly, information may need to travel through:

• Multiple Transformer layers
• Global tokens
• Other long-range sparse connections

A poorly designed sparse pattern can therefore make some long-range dependencies harder to capture.

Is sparse attention the same as sliding-window attention?

No.

Sliding-window attention is one specific type of sparse attention.

Sparse attention is the broader category.

For example:

Sparse attention
├── sliding-window attention
├── block-sparse attention
├── global + local attention
└── other structured patterns

Does sparse attention mean the model ignores most of the context?

Not necessarily.

A token may not directly attend to every other token in every layer, but information can still propagate through multiple layers or through specially connected positions.

The effectiveness depends on how the sparse pattern is designed.

Is sparse attention always better than full attention?

No.

Full attention provides direct interaction between all allowed token pairs, which can be useful for complex long-range relationships.

Sparse attention trades some of that flexibility for better efficiency.

For the local version, see “What is sliding-window attention?”

In short: sparse attention reduces the number of token-to-token attention connections by allowing only selected interactions. This can make long-context processing more efficient, but the attention pattern must be designed carefully so important information can still flow across the sequence.