What is a Vision Transformer (ViT)?

A Vision Transformer (ViT) is a neural network that uses a Transformer encoder to recognize images. It treats an image as a sequence of small patches, allowing self-attention to relate information from different parts of the image.

How does ViT work?

• Patches: Divide an image into equal-sized, non-overlapping regions.
• Tokens and positions: Represent each patch as a token and add information about where it came from. The mechanics of patch embeddings are explained separately.
• Transformer encoder: Multi-head self-attention connects patches, and feed-forward layers refine their representations.
• Classification: In the original ViT design, a learned [CLS] token summarizes the image, and a classification head predicts its class.

Practical example

Consider a 224 × 224 photograph processed in 16 × 16 patches.

• 224 ÷ 16 = 14 patches along each side.
• 14 × 14 = 196 image patches.
• With a [CLS] token, the encoder receives 197 tokens.

For example, to recognize a dog, self-attention can connect a patch containing an ear with patches containing the face and body, even when they are far apart.

Why does it matter?

• Self-attention provides a way to combine information across distant image regions.
• Unlike a traditional CNN, the basic ViT does not use stacked sliding convolution filters as its main feature extractor.
• Trade-offs: ViT often benefits from large-scale pretraining. More patch tokens also make standard self-attention more computationally expensive.

Remember: ViT describes the full image-understanding architecture; how its patch tokens are formed is a separate topic.

Research: An Image is Worth 16x16 Words — Dosovitskiy et al. (ICLR 2021).