What is a projection head in self-supervised deep learning?

A projection head is a small learnable network placed after an encoder during self-supervised training. The encoder produces an image representation h; the projection head transforms it into another vector z, which is used to calculate a training objective such as contrastive loss.

Simple flow: Image → Encoder → h → Projection head → z → Training loss

The distinction matters because the representation used for learning the loss need not be the best representation to reuse later.

Encoder representation versus projection space

• Encoder output `h = f(x)`: The features extracted from image x, intended to capture useful visual information.
• Projection output `z = g(h)`: A transformed representation in projection space, where the self-supervised loss is applied.
• Training: The loss on z sends gradients backward through the projection head and into the encoder, so both are learned together.
• Downstream use: In a common contrastive-pretraining workflow, discard the projection head and reuse h for classification or another vision task. Some methods use z or retain/adapt the projector, so discarding it is a convention, not an absolute rule.

Beginner-friendly numerical example

Suppose an encoder processes an unlabeled image and produces a two-number embedding:

h = [2, 1]

For teaching, use a linear projection head with no bias:

z = Wh, with W = [[1, −1], [0, 2]]

Step 1 — Calculate the first projected value.

z₁ = 1×2 + (−1)×1 = 1

Step 2 — Calculate the second projected value.

z₂ = 0×2 + 2×1 = 2

Step 3 — Combine them into the projection vector.

z = [1, 2]

Step 4 — Optionally normalize for a cosine-similarity-based contrastive loss.

The vector length is ||z|| = √(1² + 2²) = √5 ≈ 2.236.

z_normalized = [1/√5, 2/√5] ≈ [0.447, 0.894]

What changed? The encoder embedding was h = [2, 1], but the projected embedding used for the loss is z = [1, 2] (or its normalized version). This is a deliberately simple numerical illustration: a real head learns its weights, and different networks may output vectors with different dimensions.

Linear versus MLP projection heads

• Linear head: Applies a transformation such as z = Wh + b. It can rotate, combine, rescale, or change the dimensions of features, but it cannot express arbitrary nonlinear transformations.
• MLP head: Uses two or more learnable layers, usually with nonlinear activations between them. A simple two-layer form is z = W₂ ReLU(W₁h + b₁) + b₂.
• Why try an MLP? Research on visual contrastive learning showed that a learnable nonlinear transformation between the encoder representation and the loss can produce more useful downstream encoder representations than applying the loss directly to h. This is an empirical finding, not a promise that every MLP will outperform every linear head.

The projector can have a smaller output dimension than the encoder, but that is a design choice, not the definition of a projection head.

Why does the extra head help?

• Separate objectives: The projection space can specialize in satisfying the self-supervised contrastive objective, while the earlier encoder features may retain information helpful for other tasks.
• Protect transferable information: Some attributes useful for downstream tasks may be unnecessary for matching transformed views. A learned projection can reduce pressure for every encoder feature to be directly optimized in the final contrastive space.
• Improve representation quality in some setups: Experimental and later theoretical work have investigated why retaining the pre-projection features often benefits downstream evaluation.

Important: A projection head does not guarantee better transfer. Its value depends on the encoder, the loss, training data, architecture, and evaluation task.

What happens after training?

1. Pretrain: Optimize the encoder and projection head together using unlabeled images.
2. Save the encoder: Keep the learned feature extractor f; commonly remove the projector g.
3. Evaluate or adapt: Attach a linear classifier to frozen `h` features (a *linear probe*), or fine-tune the encoder on labeled examples.
4. Compare fairly: Measure downstream accuracy or retrieval quality with and without a projection head, holding other experimental choices as constant as possible.

Limitations and practical considerations

• Extra parameters and compute: An MLP adds layers, training work and memory use, although it is usually smaller than the backbone encoder.
• Information loss: A narrow or poorly designed projection may discard useful information from z; this is one reason the pre-projection `h` is often evaluated downstream.
• Architecture choice: Depth, hidden width, output dimension, normalization and activation can influence stability and transfer performance. More complicated is not automatically better.
• Downstream mismatch: The best training projection space may not be the best inference feature space. Test on the actual target task rather than selecting a projector only because contrastive loss decreases.

For the image-view choices feeding the encoder, see “Why are data augmentations important in contrastive visual learning?” (Question 266). For the loss and temperature, see Questions 264–265. Question 268 separately covers the full SimCLR training framework; this article focuses only on the projection-head component.

Key takeaway: The encoder creates transferable features `h`, while the projection head maps them into `z` for self-supervised training. The head helps separate the loss-optimized space from the representation often reused downstream.

Research: Chen et al. (ICML 2020), *A Simple Framework for Contrastive Learning of Visual Representations*; Xue et al. (ICLR 2024), *Investigating the Benefits of Projection Head for Representation Learning*.