What is attention ?

Attention is a mechanism that lets a model assign different importance to different parts of its input when producing an output.

Instead of treating every input element equally, attention asks: which pieces of information are most relevant right now?

For example, in a sentence, when the model processes one word, it can give higher attention to other words that provide useful context, even if they are far away in the sequence.

How attention works

A common form of attention uses three representations:

• Query (Q): what the current position is looking for
• Key (K): what each input position can be matched against
• Value (V): the information carried by each position

The model first measures how well each query matches the keys. Those scores are converted into attention weights, and the values are combined using those weights.

For scaled dot-product attention:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

Here, dₖ is the dimension of the key vectors. Dividing by √dₖ keeps the scores from becoming too large before the softmax operation.

Simple intuition

Suppose a model is processing:

The cat sat on the mat because it was tired.

When interpreting “it”, the model may assign a larger attention weight to “cat” than to less relevant words such as “mat”.

This allows the representation of “it” to incorporate information from the more relevant part of the sentence.

Why attention is useful

Attention helps models:

• capture relationships between distant parts of an input
• focus dynamically on relevant information
• build context-aware representations
• process sequence elements more flexibly than relying only on nearby information

When queries, keys, and values all come from the same sequence, the mechanism is called self-attention. Self-attention is a core building block of Transformers. See “What is a Transformer?” for the broader architecture.

In short, attention lets a model decide what information matters most for the current computation and combine that information using learned relevance weights.