What is a Transformer?

A Transformer is a neural-network architecture designed to process sequences by using attention rather than relying on recurrence.

Core idea

For each token, the model creates three learned representations: query (Q), key (K), and value (V). Self-attention compares queries with keys to determine how strongly tokens should attend to one another, then combines the corresponding values.

A common form is:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V

This lets the model connect information across a sequence directly. For example, in the sentence “The animal didn't cross the street because it was tired,” attention can help the representation of “it” incorporate information from “animal.”

Main building blocks

A Transformer typically combines:

• Token embeddings to represent input tokens as vectors.
• Positional information so token order is represented.
• Multi-head self-attention so different attention heads can learn different relationships.
• Feed-forward networks that transform each token representation.
• Residual connections and normalization that help train deep networks.

Encoder and decoder variants

The original Transformer contains both an encoder and a decoder. Modern models often use only part of that design:

• Encoder-only models are commonly used for understanding or representation tasks.
• Decoder-only models generate tokens autoregressively and are widely used for LLMs.
• Encoder-decoder models are useful when transforming one sequence into another, such as translation.

Why Transformers matter

Unlike recurrent neural networks, Transformers can process many sequence positions in parallel during training. Self-attention also provides a direct mechanism for modeling relationships between distant tokens.

The tradeoff is that standard self-attention requires computing interactions between pairs of sequence positions, giving it roughly quadratic attention cost with sequence length. This becomes important when designing systems for long contexts.

In short, a Transformer repeatedly uses attention and feed-forward computation to build context-aware token representations. This architecture became the foundation for many modern language models and has also been adapted to vision, audio, and multimodal systems.