Deep Learning
Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.
What is a pretext task in self-supervised visual learning?
A pretext task is a training problem created automatically from unlabeled images so a neural network can learn useful visual features before tackling a real application. The goal is not necessarily to become excellent at the pretext task itself; it is to learn an image representation that can be reused for another task. How does a pretext task work? Start with unlabeled images: Collect images without manually assigned object categories such as “cat,” “dog,” or “car.” Create your own supervision: Apply a known transformation or extract a relationship already present in an image. The transformation or relationship provides an automatically generated target , sometimes called a pseudo label . Train an encoder: Ask the network to solve that task. To succeed, it may need to learn edges, object structure, spatial relationships, and other visual patterns. Reuse the learned features: Keep the encoder (backbone) for a downstream task , often discarding the temporary pretext task prediction head. This is self supervised learning because the supervision comes from the data and its transformation—not from a person manually labeling image content. For the broader distinction among learning paradigms, see “What is the difference between supervised, unsupervised, and self supervised deep learning?” . Beginner friendly practical example: Predict an image's rotation Imagine you have 100 unlabeled photographs and want to learn useful image features before building a cat versus dog classifier. Step 1 — Generate targets automatically. For each original image, make four versions and record the rotation you applied: 0° rotation → class 0 90° rotation → class 1 180° rotation → class 2 270° rotation → class 3 One original photo now gives four labeled training examples . Your 100 unlabeled photos produce 100 × 4 = 400 rotation labeled examples , without anyone identifying the objects inside them. Step 2 — Train the rotation predictor. Feed a rotated image to a visual encoder and a temporary four class rotation head . For a picture rotated 90° , the target is class 1 , or a one hot vector [0, 1, 0, 0]. Suppose the network predicts rotation probabilities [0.05, 0.80, 0.10, 0.05]. Its cross entropy loss for the correct class is: L = −ln(0.80) ≈ 0.223 The model receives a real training signal, even though the image's object category is unknown . Step 3 — Transfer the representation to the real task. After rotation pretraining, discard the four class rotation head. Keep the encoder and attach a new classifier for cat versus dog . Train or fine tune that classifier using the smaller set of available human labeled cat/dog examples ; alternatively, freeze the encoder and train only a simple classifier on its features. Why could this help? Predicting orientation may encourage the encoder to notice useful shapes and part arrangements. But good rotation accuracy is not proof that the learned features will classify cats and dogs well: that must be checked on held out downstream data. Other examples of visual pretext tasks Jigsaw puzzle: Divide an image into shuffled patches; predict their correct spatial arrangement to encourage understanding of parts and structure. Relative patch position: Take two crops from one image and predict where one lies relative to the other. Image colorization: Remove color from an image and ask the model to predict plausible colors using information from the original data. Predict missing visual content: Hide part of an input and learn from reconstruction targets provided by the image itself. Detailed masked image methods are covered later in the curriculum. These are examples of pretext objectives , not a requirement that every modern self supervised method use a hand designed rotation or jigsaw classifier. Pretext task vs. downstream task Pretext task: A deliberately constructed learning challenge, such as “How was this image rotated?” The answer can be generated automatically. Downstream task: The actual goal, such as “Is there a dog in this photo?” , “Where is the object?” , or “Which pixels belong to a road?” It may use labeled examples for fine tuning or evaluation. Success criterion: Pretext task performance is useful for diagnosing pretraining, but downstream quality and label efficiency determine whether the representation is valuable. Benefits, limitations, and use cases Benefit: Lets a model exploit large collections of unlabeled images, potentially reducing how many costly human annotations are needed for a later task. Applications: Pretraining for image classification, object detection, image segmentation, and specialized domains where labels are scarce. Task mismatch: A network can learn shortcuts such as image borders or rotation artifacts instead of transferable object semantics. Some datasets have no meaningful “upright” orientation, and symmetric objects make rotation labels ambiguous. Not guaranteed: A poorly chosen pretext task can yield weak representations. Compare against training from scratch and evaluate on the actual downstream dataset . Key takeaway: A pretext task creates its own targets from unlabeled images so the encoder can learn transferable visual features; the downstream task determines whether those features are genuinely useful. The next question introduces contrastive representation learning as a separate self supervised learning principle, without repeating the modern methods that follow later. Research: Gidaris, Singh & Komodakis (2018), Unsupervised Representation Learning by Predicting Image Rotations ; Noroozi & Favaro (2016), Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles ; Doersch, Gupta & Efros (2015), Unsupervised Visual Representation Learning by Context Prediction .
What is contrastive representation learning in deep neural networks?
Contrastive representation learning teaches a neural network to turn inputs into useful numerical representations (embeddings) by making related views more similar and contrasting views less similar. In self supervised visual learning, those relationships can be generated from unlabeled images, so the encoder can learn before it ever sees human provided cat, dog, or car labels. What is the objective? A visual encoder f(x) transforms an image x into an embedding h = f(x)—a vector of numbers describing visual features. Bring related views together: If we crop or recolor the same source photo in two reasonable ways, we want their learned representations to be similar. Contrast with other views: Representations of other sampled source images act as contrasting examples, helping the network avoid producing the same embedding for everything. Learn transferable information: To match different views reliably, the encoder may learn object parts, shape, texture, and other features that survive those transformations. The training goal is to increase similarity for related representations while decreasing it for contrasting representations . A contrastive loss supplies gradients for this relative comparison. The exact construction of positive and negative pairs is Question 263; the mathematical loss and its temperature are covered in Questions 264–265. Beginner friendly image example Suppose you have two unlabeled images : a photograph of a dog sitting in a park and a photograph of a bicycle. 1. Create two views of the dog image: Make a cropped view and a brightness adjusted view. Both originate from the same photo , so their relationship is known automatically. 2. Encode the views: Pass each version through the same visual encoder to produce compact embeddings. 3. Encode a contrasting view: Pass the bicycle image through that encoder too. 4. Train the encoder: Encourage the two dog photo views to have more similar embeddings than the dog and bicycle views. Repeat over many images and batches. No one needed to label either source photo. The model learns which views belong together , not a direct “dog versus bicycle” classification rule. Tiny numerical example: Comparing learned embeddings Imagine the encoder produces these illustrative, two dimensional, unit length embeddings : Anchor (first dog view): a = [1, 0] Related view (second view of the same dog photo): b = [0.8, 0.6] Contrasting image (bicycle): c = [0, 1] A common measure is cosine similarity , the dot product divided by the vectors' lengths: sim(u, v) = (u · v) / (||u|| × ||v||) All three example vectors have length 1, so the arithmetic is especially simple: Anchor versus related view: sim(a, b) = 1×0.8 + 0×0.6 = 0.8. Anchor versus contrasting image: sim(a, c) = 1×0 + 0×1 = 0. Interpretation: The related views already have higher similarity (0.8) than the contrasting views (0) . The training objective encourages this ordering for the sampled comparisons. These are made up embeddings for explanation , not measured outputs or a guarantee that a trained network will produce those exact numbers. In practice, embeddings have many more dimensions, and the loss compares multiple image views. Why not just make every embedding identical? If every image mapped to the same vector, the two views of each image would indeed match—but the representation would tell us nothing useful about differences between images. Contrasting examples help prevent that trivial collapse by requiring the encoder to distinguish views from different sources. This is also why the definition of “related” matters: transformations should preserve the visual information that the downstream application will need. What can the learned representations be used for? After self supervised training, a practitioner can freeze the encoder and train a small labeled classifier on its features, or fine tune the encoder using the labeled dataset. Image classification: Reuse learned features when manually labeled images are limited. Visual similarity search: Retrieve photographs whose embeddings are nearby, subject to the representation's learned notion of similarity. Detection or other vision tasks: Use the pretrained encoder as an initialization and adapt it for the task. The important output is a useful encoder , not necessarily the ability to identify a class directly during contrastive pretraining. Benefits and limitations Benefits: Learns from large unlabeled datasets, can improve label efficiency, and may transfer to several downstream vision tasks. False contrasts: Different source photos may depict the same object category or even the same object ; treating them as unrelated can provide a misleading training signal. Augmentation sensitivity: Excessively strong cropping or color changes can remove essential information, while weak changes may lead to superficial matching. Resource cost: Comparing many views can require substantial compute and memory; the effectiveness depends on sampling and the chosen objective. Not guaranteed: Higher contrastive training performance does not by itself prove that downstream classification or retrieval improves. Evaluate with held out downstream data. For the general idea of learning from self generated supervision, see “What is a pretext task in self supervised visual learning?” (Question 261). This article introduces the representation learning objective ; later questions explain pair selection, loss formulas, temperature, and individual training frameworks separately. Key takeaway: Contrastive learning builds an embedding space in which related image views are encouraged to be closer than contrasting views, producing features that may transfer to other visual tasks. Research: Chen et al. (ICML 2020), A Simple Framework for Contrastive Learning of Visual Representations ; Wu et al. (CVPR 2018), Unsupervised Feature Learning via Non Parametric Instance Discrimination ; Le Khac et al. (2020), Contrastive Representation Learning: A Framework and Review .
What are positive and negative pairs in contrastive learning?
Positive and negative pairs tell a contrastive learning model which image views should be treated as related and which should be distinguished. A positive pair is usually two augmented views of the same source image . A negative pair is usually a view of that image paired with a view of a different source image . These relationships can be created without human class labels. What are the three roles? Anchor: The image view whose learned representation we are comparing with other views. Positive: A related view—for example, another crop or brightness adjusted version of the anchor's original photograph. We want its embedding to remain similar to the anchor's. Negative: A contrasting view, typically from another original photograph. The model is trained to distinguish it from the anchor. Important distinction: In common instance discrimination training, “different images” does not necessarily mean “different semantic classes.” Two photos can both show bicycles yet be assigned a negative relationship because they come from different original images. Beginner friendly example: Selecting image pairs Suppose our dataset contains three unlabeled source photos: Photo A: A red bicycle beside a shop. Photo B: A red motorcycle on a road. Photo C: A green tree. For Photo A , create two random but meaning preserving views: a crop and a slightly color adjusted crop. Anchor: First view of Photo A. Positive: Second view of Photo A —same underlying photo, despite visual changes. Harder negative: A view of Photo B , since the red motorcycle may share visual features with the red bicycle. Easier negative: A view of Photo C , a tree with quite different visual content. The relationships were created by tracking which original photo each view came from . Nobody had to manually label the pictures as bicycles, motorcycles or trees. Tiny numerical example: How similar are these pairs? Imagine an encoder has produced the following illustrative, unit length 2D embeddings : Anchor: a = [0.6, 0.8] Positive: p = [0.8, 0.6] Hard negative: n h = [1, 0] Easy negative: n e = [−0.8, 0.6] For unit length vectors, cosine similarity is their dot product. 1. Anchor–positive: sim(a,p) = 0.6×0.8 + 0.8×0.6 = 0.96. 2. Anchor–hard negative: sim(a,n h) = 0.6×1 + 0.8×0 = 0.60. 3. Anchor–easy negative: sim(a,n e) = 0.6×(−0.8) + 0.8×0.6 = 0.00. Interpretation: The positive is very similar ( 0.96 ); the hard negative is more similar (0.60) to the anchor than the easy negative ( 0.00 ), so it is harder to distinguish. These numbers are invented for teaching , not empirical measurements or guaranteed outcomes. Training can change all of these similarities. Where do in batch negatives come from? An efficient strategy is to create two views per source image in a training batch. For N = 3 source photos: The batch contains 2N = 6 augmented views. Choose one anchor view: Exclude the anchor itself. One corresponding positive: The other view of the same original photo . Remaining contrasting views: The other 2N − 2 = 4 views come from the two different source photos and can serve as in batch negatives . They are called in batch because the other examples already in the same training batch are reused as contrasts; no separate hand labeled negative dataset is necessary. This count describes the two view, one positive instance discrimination setup , not every contrastive learning method. More negatives may increase compute and may increase semantic collisions; more is not automatically better. What are false negatives and hard negatives? False negative: Two views come from different source images but are actually semantically related for the downstream task . For example, two separate pictures of bicycles may be labeled negative by source identity, even though a future bicycle category classifier should regard them as similar. Hard negative: A genuinely contrasting example whose current embedding is unusually similar to the anchor . A bicycle and a motorcycle might be hard to distinguish because of shared shapes and wheels. The risk: A highly similar image chosen as a “hard negative” can actually be a false negative . Aggressively pushing it away may damage the semantics the model should preserve. Research on false negative cancellation and hard negative sampling studies ways to identify or reduce these issues. There is no guarantee that simply mining the closest negatives improves representations. How do you choose pairs responsibly? Preserve meaning in positives: A crop should not accidentally remove the relevant object or turn the view into something fundamentally different. Poorly chosen augmentations can make false positive relationships . Avoid known duplicates or semantic collisions: Where metadata or labels are available, they can help detect cases in which “different filenames” do not imply distinct content. Be cautious with hard negative mining: Similarity may reflect true shared semantics, not merely a good training challenge; use appropriate filtering and check downstream quality. Use the right relationship for the objective: In supervised contrastive learning , images with the same known class can be treated as positives. In common self supervised instance discrimination, source image identity provides the relationship without those labels. Evaluate on real tasks: Check whether the learned encoder improves retrieval, classification or transfer—not just whether pair similarities separate in training. Key takeaway: Positive pairs normally link different views of one source image; negative pairs normally compare different sources. Hard negatives can provide useful challenges, but false negatives and poorly selected augmentations can corrupt the learning signal. For the overall purpose of learning embeddings, see “What is contrastive representation learning in deep neural networks?” (Question 262). The InfoNCE loss and temperature parameter are covered in Questions 264 and 265; their formulas are deliberately left out here. Research: Chen et al. (ICML 2020), A Simple Framework for Contrastive Learning of Visual Representations ; Huynh et al. (WACV 2022), Boosting Contrastive Self Supervised Learning With False Negative Cancellation ; Robinson et al. (ICLR 2021), Contrastive Learning With Hard Negative Samples .
What is InfoNCE loss in contrastive learning?
InfoNCE (Information Noise Contrastive Estimation) is a contrastive loss that teaches a neural network encoder to give a related (positive) view a higher score than competing negative views. It assigns a softmax score to the correct positive and penalizes the encoder when that score is low. How is InfoNCE calculated? For one anchor , one positive and K negative candidates, a common form is: L InfoNCE = −ln[ exp(s pos/τ) / (exp(s pos/τ) + Σ k exp(s neg,k/τ)) ] s pos: Similarity between the anchor and positive representation, often cosine similarity. s neg,k: Similarity between the anchor and its kth negative. τ 0: The temperature, which scales similarity scores before exponentiation. We hold it fixed here; its effects are covered in Question 265. Numerator: exp(s pos/τ), the positive's exponentiated score. Denominator: The positive's exponentiated score plus the exponentiated scores of all negatives . The positive must be included. Outer −ln: Takes the negative natural logarithm of the score assigned to the positive, making the loss small when the model ranks the positive confidently. The anchor itself is excluded from the candidate choices. This formula expresses a single anchor loss, not necessarily the full training batch average. Beginner friendly step by step numerical example Imagine one anchor image, a positive second view of that same original photo, and two negatives . Suppose the encoder gives these illustrative cosine similarities: Positive: s pos = 0.8. Negative 1: s neg,1 = 0.4. Negative 2: s neg,2 = 0.0. Temperature: τ = 0.2. Step 1 — Scale the similarity scores. Positive: 0.8 / 0.2 = 4. Negative 1: 0.4 / 0.2 = 2. Negative 2: 0 / 0.2 = 0. Step 2 — Take the exponential of each value. Positive: exp(4) ≈ 54.5982. Negative 1: exp(2) ≈ 7.3891. Negative 2: exp(0) = 1.0000. Step 3 — Add the three exponentials for the denominator. D = exp(4) + exp(2) + exp(0) ≈ 62.9872 Notice that the numerator's positive term also appears in the denominator . Step 4 — Calculate the score assigned to the correct positive. p positive = exp(4) / D ≈ 0.8668 (about 86.68%). This is the softmax probability among these three candidate views that the designated positive is the correct match. It is not a probability that the image belongs to a particular object class . Step 5 — Calculate the InfoNCE loss. L InfoNCE = −ln(0.8668) ≈ 0.1429 Answer: For this anchor and these two negatives, the InfoNCE loss is approximately 0.1429 . The similarity scores are invented for a numerical illustration, not measured experimental results. Why do the negative examples matter? Competition: Each negative adds a positive term to the denominator. If a negative's similarity rises while the positive similarity stays unchanged, the positive probability falls and the loss increases. No negatives: In this simple formula, removing every negative makes numerator equal denominator. Then p positive = 1 and the loss is 0 for any score: there is no meaningful contrastive training signal. False negatives: A “negative” may accidentally depict the same semantic category as the anchor; forcing it away can hurt representation quality. Pair construction pitfalls are covered in “What are positive and negative pairs in contrastive learning?” (Question 263). How is it used in real training? Across a batch: Compute the loss for multiple anchors and average their individual losses. Often both views of a positive pair serve as anchors. Numerically stable calculation: Use log softmax, cross entropy or log sum exp instead of directly exponentiating large logits, which could overflow. Learned representations: Gradients from the loss update the encoder to make the positive more distinguishable from the alternatives. Evaluation: A lower contrastive training loss is not automatically better downstream classification or retrieval. Test representations on held out data. Benefits: InfoNCE supplies a differentiable learning signal from image view relationships even without manually supplied object categories. Limitations: Its effectiveness depends on meaningful pair selection; using many negatives can cost memory and compute, and false negatives can distort learning. For the overall goal of learning visual embeddings, see “What is contrastive representation learning in deep neural networks?” (Question 262). The next article, “How does temperature affect contrastive learning?” (Question 265), explains what changes when τ varies; this article deliberately keeps the temperature fixed. Key takeaway: InfoNCE uses the positive's exponentiated similarity as a numerator, divides by the sum over positive and negative candidates, and minimizes the negative logarithm of the resulting positive match score. Sources: van den Oord et al. (2018), Representation Learning with Contrastive Predictive Coding ; Chen et al. (ICML 2020), A Simple Framework for Contrastive Learning of Visual Representations .
How does temperature affect contrastive learning?
Temperature (τ) controls how sharply contrastive learning distinguishes an image's positive view from competing negative views. It scales similarity scores before softmax. A lower temperature makes the distribution more concentrated on high similarity candidates; a higher temperature spreads probability more evenly. What does changing temperature do? Contrastive learning converts each similarity s into a softmax logit using s / τ, where τ 0 . Low τ: Magnifies differences between scores. Hard negatives receive a greater relative share of attention compared with easy negatives. High τ: Compresses differences. More of the candidate probability is spread across images with lower similarity. Important: Temperature affects gradient weighting and training behavior , not just the displayed softmax probabilities. Lower is not always better . For the full numerator, denominator and logarithm of InfoNCE, see “What is InfoNCE loss in contrastive learning?” (Question 264). Here we hold all candidate similarities fixed and change only temperature. Numerical example: Compare three temperatures Suppose an encoder gives the following illustrative cosine similarities for one anchor: Positive similarity: 0.8 Harder negative similarity: 0.4 Easier negative similarity: 0.0 At each temperature, divide each score by τ, exponentiate, sum the three results, then compute the positive softmax probability and InfoNCE loss = −ln(p positive) . 1. Low temperature: τ = 0.1 Scaled scores: [0.8/0.1, 0.4/0.1, 0/0.1] = [8, 4, 0]. Exponentials: [2980.9580, 54.5982, 1.0000]. Denominator: 2980.9580 + 54.5982 + 1.0000 ≈ 3036.5561. Positive probability: 2980.9580 / 3036.5561 ≈ 0.9817. InfoNCE loss: −ln(p positive) ≈ 0.0185. 2. Medium temperature: τ = 0.2 Scaled scores: [4, 2, 0]. Exponentials: [54.5982, 7.3891, 1.0000]. Denominator: 54.5982 + 7.3891 + 1.0000 ≈ 62.9872. Positive probability: 54.5982 / 62.9872 ≈ 0.8668. InfoNCE loss: −ln(p positive) ≈ 0.1429. 3. Higher temperature: τ = 0.5 Scaled scores: [1.6, 0.8, 0]. Exponentials: [4.9530, 2.2255, 1.0000]. Denominator: 4.9530 + 2.2255 + 1.0000 ≈ 8.1786. Positive probability: 4.9530 / 8.1786 ≈ 0.6056. InfoNCE loss: −ln(p positive) ≈ 0.5015. What changed? For the same scores , the positive probability grows from 60.56% (τ=0.5) to 98.17% (τ=0.1), while the example loss drops from 0.5015 to 0.0185 . This does not prove that 0.1 is a better temperature for training: these probabilities are conditioned on a fixed set of candidates, not a measure of downstream accuracy. How does temperature affect hard negatives and gradients? Hard negative sensitivity: Here, the harder negative has similarity 0.4, the easier one 0.0. At τ=0.1, about 98.20% of the total negative only softmax mass belongs to the harder one; at τ=0.5, the share is about 69.00% . Lower temperature emphasizes the harder negative relative to the easy one . Gradient scaling: For this one positive InfoNCE loss, ∂L/∂s negative = p negative/τ, while ∂L/∂s positive = (p positive − 1)/τ. The encoder gradients additionally depend on how the embeddings affect similarities. Saturation: The absolute derivative for the harder negative is approximately 0.1798 at τ=0.1 but 0.5866 at τ=0.2. Once the softmax is highly confident, its gradients can become small even with low temperature. Relative emphasis is not the same as absolute gradient strength. False negatives: Incorrectly contrasting two semantically related images can become especially harmful when temperature heavily penalizes similar negatives. How should temperature be selected? Validate rather than guess: Start from a reasonable baseline for your model and objective; evaluate several positive temperature settings and compare downstream classification or retrieval quality . Monitor stability: Very low τ may make gradients sensitive to difficult/false negatives or produce saturated logits; very high τ may make contrast insufficiently discriminative. Use stable computation: Implement InfoNCE with log softmax or log sum exp to avoid overflow for large scaled logits. Do not confuse hyperparameters: Temperature controls contrastive loss score scaling ; the optimizer's learning rate controls parameter update size . They are not interchangeable. Key takeaway: Lower temperature concentrates probability on high similarity candidates and relatively emphasizes hard negatives; higher temperature softens the distribution. The best setting depends on training stability and downstream quality, not on the loss of one fixed example. Research: Wang & Liu (CVPR 2021), Understanding the Behaviour of Contrastive Loss ; Chen et al. (ICML 2020), A Simple Framework for Contrastive Learning of Visual Representations .
Why are data augmentations important in contrastive visual learning?
Data augmentations are important in contrastive visual learning because they create different, meaningful views of an unlabeled image and teach an encoder which visual changes should not alter its representation. When chosen well, they encourage the model to recognize an object despite changes in crop, lighting or image appearance—not just memorize pixel details. Why do augmentations matter? Learn useful invariance: An invariant representation stays relatively similar when an image undergoes a change that should not matter for the intended task. For instance, a car should still look like a car under modest brightness changes. Make learning challenging enough: If two views are nearly identical, the network may match them using unhelpful shortcuts such as colors or low level pixel patterns. Avoid making learning impossible: If a transformation removes the object or changes a label defining feature, encouraging two views to match may teach the wrong invariance . Define what the encoder learns: Augmentation choices are part of the training objective, not merely cosmetic preprocessing. Beginner friendly practical example Imagine an unlabeled 256 × 256 pixel photograph of a car parked beside a tree. Create two random views independently: 1. View A: Crop and resize a region containing the whole car , keeping roughly 80% of the original image area. Apply a small brightness change. 2. View B: Crop and resize a different region that still contains the car , keeping roughly 70% of the original image area. Apply a horizontal flip and mild blur. 3. Train: Send both views through the encoder. The contrastive objective encourages similar representations despite the different crops, brightness and blur. 4. What it might learn: The car's shape, parts and structure remain recognizable when background and lighting change. This could help later image classification when labeled data are limited. Important: The 80% and 70% crop areas are illustrative , not recommended fixed settings. Random crops are not guaranteed to preserve the car; an actual pipeline must choose suitable crop ranges and check its samples. What do common visual augmentations teach? Random crop and resize: Encourages robustness to changes in visible region, object position and scale. Risk: An aggressive crop may leave only the tree—or remove a small, important object. Color jitter: Changes brightness, contrast, saturation or hue. Encourages attention to features beyond exact colors. Risk: If the task is to distinguish red cars from blue cars , strong color jitter destroys information the classifier needs. Horizontal flip: Can help when left facing and right facing versions should mean the same thing. Risk: It may change the meaning of text, traffic signs, left versus right labels or other orientation sensitive images. Gaussian blur: Reduces fine visual details and can encourage attention to broader shapes. Risk: Too much blur hides tiny features such as defects, lesions or fine textures. These transforms are typically applied randomly and in combination . The correct policy depends on the domain: natural photographs, medical scans, satellite images and document images need different assumptions about what may safely change. How strong should augmentations be? Too weak: Views can be matched through easy shortcuts rather than transferable features. Research on visual contrastive learning found that combining random crops with color distortion was especially useful: without color changes, the network could rely heavily on color statistics shared by crops. Appropriately strong: Views differ enough to require useful visual understanding while preserving the information needed later . Too strong: Views can lose their common meaning, erase small objects or enforce invariance to an important feature. This can harm downstream accuracy even when the training objective appears to improve. Invariance is task dependent . A transformation useful for generic object classification may be harmful for color recognition, medical diagnosis or orientation prediction. No single augmentation policy is universally best. Practical tuning and limitations Inspect sample views: Save or display pairs from the augmentation pipeline. Check whether intended objects or diagnostic details remain visible. Start with modest settings: Try crop and resize plus mild color changes; add flip and blur only when they make sense for the domain. Compare policies: Vary one choice at a time—crop range, jitter strength, flip probability or blur probability—and evaluate the resulting downstream classification or retrieval quality. Watch for shortcuts and distribution mismatch: Augmentations can either suppress shortcuts or introduce unnatural artifacts. Training views should remain plausible for the target use case. Measure the tradeoff: More aggressive augmentation can increase task difficulty and processing cost without necessarily improving transferable representations. For how related and contrasting views are designated, see “What are positive and negative pairs in contrastive learning?” (Question 263). The effect of temperature is covered in Question 265. Later articles explain the components of individual self supervised frameworks. Key takeaway: Augmentation quality determines which visual differences the encoder learns to ignore. Good augmentations create useful variation while preserving task relevant meaning; overly weak or destructive changes can undermine the representation. Research: Chen et al. (ICML 2020), A Simple Framework for Contrastive Learning of Visual Representations ; Xiao et al. (2020), What Should Not Be Contrastive in Contrastive Learning .
What is a projection head in self-supervised deep learning?
A projection head is a small learnable network placed after an encoder during self supervised training. The encoder produces an image representation h; the projection head transforms it into another vector z, which is used to calculate a training objective such as contrastive loss. Simple flow: Image → Encoder → h → Projection head → z → Training loss The distinction matters because the representation used for learning the loss need not be the best representation to reuse later . Encoder representation versus projection space Encoder output h = f(x): The features extracted from image x, intended to capture useful visual information. Projection output z = g(h): A transformed representation in projection space , where the self supervised loss is applied. Training: The loss on z sends gradients backward through the projection head and into the encoder , so both are learned together. Downstream use: In a common contrastive pretraining workflow, discard the projection head and reuse h for classification or another vision task. Some methods use z or retain/adapt the projector, so discarding it is a convention, not an absolute rule. Beginner friendly numerical example Suppose an encoder processes an unlabeled image and produces a two number embedding: h = [2, 1] For teaching, use a linear projection head with no bias: z = Wh, with W = [[1, −1], [0, 2]] Step 1 — Calculate the first projected value. z₁ = 1×2 + (−1)×1 = 1 Step 2 — Calculate the second projected value. z₂ = 0×2 + 2×1 = 2 Step 3 — Combine them into the projection vector. z = [1, 2] Step 4 — Optionally normalize for a cosine similarity based contrastive loss. The vector length is ||z|| = √(1² + 2²) = √5 ≈ 2.236. z normalized = [1/√5, 2/√5] ≈ [0.447, 0.894] What changed? The encoder embedding was h = [2, 1], but the projected embedding used for the loss is z = [1, 2] (or its normalized version). This is a deliberately simple numerical illustration: a real head learns its weights, and different networks may output vectors with different dimensions. Linear versus MLP projection heads Linear head: Applies a transformation such as z = Wh + b. It can rotate, combine, rescale, or change the dimensions of features, but it cannot express arbitrary nonlinear transformations. MLP head: Uses two or more learnable layers, usually with nonlinear activations between them. A simple two layer form is z = W₂ ReLU(W₁h + b₁) + b₂. Why try an MLP? Research on visual contrastive learning showed that a learnable nonlinear transformation between the encoder representation and the loss can produce more useful downstream encoder representations than applying the loss directly to h. This is an empirical finding, not a promise that every MLP will outperform every linear head. The projector can have a smaller output dimension than the encoder, but that is a design choice, not the definition of a projection head. Why does the extra head help? Separate objectives: The projection space can specialize in satisfying the self supervised contrastive objective, while the earlier encoder features may retain information helpful for other tasks. Protect transferable information: Some attributes useful for downstream tasks may be unnecessary for matching transformed views. A learned projection can reduce pressure for every encoder feature to be directly optimized in the final contrastive space. Improve representation quality in some setups: Experimental and later theoretical work have investigated why retaining the pre projection features often benefits downstream evaluation. Important: A projection head does not guarantee better transfer. Its value depends on the encoder, the loss, training data, architecture, and evaluation task. What happens after training? 1. Pretrain: Optimize the encoder and projection head together using unlabeled images. 2. Save the encoder: Keep the learned feature extractor f; commonly remove the projector g. 3. Evaluate or adapt: Attach a linear classifier to frozen h features (a linear probe ), or fine tune the encoder on labeled examples. 4. Compare fairly: Measure downstream accuracy or retrieval quality with and without a projection head, holding other experimental choices as constant as possible. Limitations and practical considerations Extra parameters and compute: An MLP adds layers, training work and memory use, although it is usually smaller than the backbone encoder. Information loss: A narrow or poorly designed projection may discard useful information from z; this is one reason the pre projection h is often evaluated downstream. Architecture choice: Depth, hidden width, output dimension, normalization and activation can influence stability and transfer performance. More complicated is not automatically better. Downstream mismatch: The best training projection space may not be the best inference feature space. Test on the actual target task rather than selecting a projector only because contrastive loss decreases. For the image view choices feeding the encoder, see “Why are data augmentations important in contrastive visual learning?” (Question 266). For the loss and temperature, see Questions 264–265. Question 268 separately covers the full SimCLR training framework; this article focuses only on the projection head component. Key takeaway: The encoder creates transferable features h, while the projection head maps them into z for self supervised training. The head helps separate the loss optimized space from the representation often reused downstream. Research: Chen et al. (ICML 2020), A Simple Framework for Contrastive Learning of Visual Representations ; Xue et al. (ICLR 2024), Investigating the Benefits of Projection Head for Representation Learning .
What is SimCLR and how does it work?
SimCLR (Simple Framework for Contrastive Learning of Visual Representations) is a self supervised method that learns useful image features without needing object category labels during pretraining. It trains a shared encoder to recognize two independently augmented views of the same image as related, using a contrastive objective. How does SimCLR work end to end? Pipeline: Unlabeled image → Two random views → Shared encoder f → h → Nonlinear projection head g → z → Normalize → Contrastive loss 1. Sample unlabeled images. Select a mini batch of N distinct source images; human provided class labels are not needed. 2. Create two views per image. Apply independently sampled random transformations to each source image. Typical operations include cropping/resizing, color distortion and blur. The views should preserve information relevant to the intended task. 3. Run the same encoder on every view. A base encoder f (a ResNet in the original paper) converts each view into a representation h = f(view). Both views share the exact same encoder weights , not two separately trained networks. 4. Project the representations. A small nonlinear MLP head g transforms each h into z = g(h). The contrastive objective is applied to these projected features, rather than directly to h. 5. Normalize and compare. Use L2 normalized projected embeddings to calculate cosine similarities, then apply the normalized temperature scaled cross entropy loss (NT Xent) across the batch. 6. Update and repeat. Average losses across all 2N anchor views, backpropagate through both g and f , and continue training on new image batches. Why this arrangement? SimCLR combines effective view generation, a shared backbone, a learnable nonlinear projection and a batch based contrastive objective in one relatively simple training system. Beginner friendly practical example: Three unlabeled photos Imagine a training batch containing three unlabeled photos : a bicycle, a dog and a tree. We name them A, B and C for explanation; these are not class labels supplied to the model . Create views: Two independent augmentations produce A1, A2, B1, B2, C1, C2. Count views: N = 3 source images give 2N = 6 training views. Encode and project: The shared encoder produces six h vectors; the same MLP converts them into six z vectors, which are normalized for the loss. Choose anchor A1: Compare it with the five other views . A2 is its one matching view; B1, B2, C1, C2 are its four in batch contrasts . Calculate losses: Every view becomes an anchor once. Thus the batch contributes 6 anchor specific losses to the average. Each of the 3 source image matches is evaluated in both directions, such as A1 → A2 and A2 → A1. General count for N source images: Total views: 2N. Candidates for one anchor: 2N − 1 (exclude the anchor itself). Matching view: 1. Other source views: 2N − 2 contrasts. Anchor specific loss terms averaged: 2N. These counts describe standard SimCLR with exactly two views per source image . They are not a statement that all contrastive approaches use these counts. How is the encoder used after training? Discard the projection head: In the original SimCLR evaluation setup, the trained MLP projection head g is removed. Keep the encoder: The learned backbone f produces representations h for new images. Evaluate or adapt: Train a linear classifier on frozen encoder features (linear probing), or fine tune the encoder with a smaller labeled dataset for classification or other visual tasks. Training uses z; downstream evaluation commonly uses h. This separation helps the encoder learn transferable visual features instead of only optimizing the final contrastive space. Why is SimCLR useful? No class labels for pretraining: Learn from large image collections before any task specific annotation. Simple architecture: The original framework uses the same encoder for both views and does not require a separate memory bank or momentum encoder . Transferable representations: Features can support downstream classification, visual retrieval or serve as a starting point for other vision tasks. The original paper reported strong ImageNet linear evaluation and label efficient fine tuning results. Limitations and practical considerations Batch size and compute needs: More contrasting examples are available with larger batches, and the original approach benefited from large batches and longer training. This can be expensive on limited GPU memory. Quality of views matters: Excessive transformations can remove the object or change semantics. A pipeline for medical or document imagery may require different augmentations from one for ordinary photographs. False negatives: Two different photographs may show the same category, yet instance discrimination training can treat them as contrasts. Not automatically a better deployed model: Measure downstream accuracy or retrieval quality, rather than trusting contrastive loss alone. For individual building blocks, see “What are positive and negative pairs in contrastive learning?” (Question 263), “What is InfoNCE loss in contrastive learning?” (Question 264), “How does temperature affect contrastive learning?” (Question 265), “Why are data augmentations important in contrastive visual learning?” (Question 266), and “What is a projection head in self supervised deep learning?” (Question 267). Question 269 introduces Momentum Contrast (MoCo) separately. Key takeaway: SimCLR learns image representations by making two augmented views of each image agree in a normalized projection space, using other views in the same batch for contrast. After training, the shared encoder supplies features for downstream tasks. Research: Chen et al. (ICML 2020), A Simple Framework for Contrastive Learning of Visual Representations ; official SimCLR project page.
What is Momentum Contrast (MoCo) and how does it work?
Momentum Contrast (MoCo) is a self supervised visual learning method that trains an image encoder using a large, reusable dictionary of contrasting image features. Its two main components are a first in, first out (FIFO) queue of encoded keys and a slowly updated momentum key encoder . Together they make it possible to compare a query with many recent negatives without requiring all those images in the current mini batch. How does the original MoCo work? 1. Start with unlabeled images. Make two independently augmented views of each source photograph: a query view and a key view . 2. Encode the query. The query encoder f q, with parameters θ q, turns the first view into a vector q. The query encoder is trained using backpropagation. 3. Encode the matching key. The key encoder f k, with parameters θ k, turns the other view into a positive key k⁺. It starts as a copy of the query encoder but is not directly updated by contrastive loss gradients . 4. Look up negatives. Compare q with k⁺ and with stored key embeddings in the FIFO queue. Those queued entries come from previous mini batches , so the dictionary can be much larger than the current batch. 5. Apply the contrastive loss. Encourage the query to match its positive key rather than the queued negatives. Update θ q via backpropagation. 6. Update the key encoder smoothly. Use a momentum/EMA update to move θ k a small fraction toward the latest θ q. The encoder that produced stored keys therefore changes gradually. 7. Refresh the dictionary. After the current loss uses the existing queue, enqueue the current mini batch's key embeddings and dequeue the oldest entries to keep queue capacity fixed. Training flow: Query view → trainable query encoder → q; Key view → momentum encoder → k⁺; q + k⁺ + queued negatives → contrastive loss. Practical example: One mini batch, a FIFO dictionary Imagine a tiny training setup with mini batch size B = 2 , but a negative key queue capacity K = 6 . Before the step , the queue contains six older key embeddings, written from oldest to newest : [k1, k2, k3, k4, k5, k6]. Current batch: Two unlabeled source photos, A and B, produce queries qA, qB and their matching keys kA⁺, kB⁺. For query qA: There is 1 current positive (kA⁺) and 6 queued negatives (k1 … k6), giving 7 candidate keys for its contrastive comparison. Query qB has its own positive and the same six queued candidates. After computing the loss , add the two newly encoded keys for A and B. To remain at capacity six, remove the oldest two entries k1, k2. New queue: [k3, k4, k5, k6, kA, kB], again in oldest to newest order. The important numbers: A batch of 2 source photos can use 6 previously stored negatives per query. Its contrastive dictionary is therefore not limited to the two current images . In practical MoCo runs the queue can be far larger (the original paper uses a dictionary of 65,536 keys in reported settings). This is a simplified queue illustration. A real implementation manages batch ordering, device synchronization and which embeddings are used in a particular training step. Numerical example: How does momentum update the key encoder? MoCo changes the key encoder with a moving average: θ k ← m × θ k + (1 − m) × θ q θ k: Current key encoder parameter. θ q: Corresponding query encoder parameter, after its gradient update. m: Momentum coefficient between 0 and 1, typically close to 1. For one illustrative scalar parameter , let the old key weight be θ k = 2.0 and the updated query weight be θ q = 4.0 . With m = 0.9: θ k,new = 0.9×2.0 + 0.1×4.0 = 2.2. With m = 0.999: θ k,new = 0.999×2.0 + 0.001×4.0 = 2.002. Even though the query weight is now 4.0 , the key weight changes only slightly. The original MoCo used a high momentum, commonly m = 0.999, to improve consistency across keys encoded at different training steps. The value 0.9 here is for illustrating the arithmetic, not a recommendation. Why use both a queue and a momentum encoder? Queue → many negatives without a huge current batch. The queue reuses key embeddings from prior steps; its capacity K is chosen separately from mini batch size B. Momentum → more comparable stored keys. Since queued keys were created at earlier steps, rapidly changing the encoder would make old and new embeddings inconsistent. Slow updates reduce this mismatch. FIFO → reduce staleness. Removing the oldest embeddings stops the queue from keeping entries indefinitely. No gradients through stored keys. Previously encoded queue entries are treated as detached features; it would be expensive and inappropriate to backpropagate through every historical image at each step. Where is MoCo useful, and what are its limitations? Downstream use: Keep the pretrained query encoder as a visual backbone, then evaluate its features with a linear classifier or fine tune for tasks such as image classification, object detection and segmentation. Advantages: Uses unlabeled data; supplies many negatives with a relatively small mini batch; avoids a costly re encoding of the entire dictionary at every step. Queue tradeoffs: The queue occupies memory, can contain stale features , and needs careful distributed synchronization. A poorly chosen length may retain outdated keys. Momentum tradeoffs: An excessively fast changing key encoder weakens consistency; excessively slow updates may lag behind the current query encoder. Data tradeoffs: False negatives and unhelpful image augmentations can still damage learned features. A large dictionary does not guarantee good downstream accuracy. Relationship to earlier lessons: Question 268 explains SimCLR , which contrasts views within the current batch. MoCo instead emphasizes a reusable queue and momentum key encoder . For generic pair construction or the loss derivation, see Questions 263 and 264; they are not repeated here. Key takeaway: MoCo learns image representations by comparing a trainable query with a matching key and many queued negatives. Its FIFO dictionary expands the contrastive set beyond the current batch, while a momentum updated key encoder keeps stored representations reasonably consistent. Research: He et al. (CVPR 2020), Momentum Contrast for Unsupervised Visual Representation Learning ; original arXiv paper.
What is representation collapse in self-supervised learning?
Representation collapse occurs when a self supervised encoder produces identical or insufficiently varied embeddings for different inputs. A model can learn to make two augmented views agree while losing the distinctions needed for classification, retrieval, and other downstream tasks. Complete, near and dimensional collapse Complete collapse: Every image maps to the same embedding vector , regardless of its content. No information about differences between images survives. Near collapse: Embeddings differ only slightly; they may look almost identical under cosine similarity and exhibit tiny variance. Dimensional collapse: The representation retains some variation, but many of its coordinates are constant or redundant, so the actual data occupy a much smaller number of independent directions than the embedding dimension suggests. Why collapse is tempting: If a training objective only rewards agreement between two views of the same input, returning one constant vector for all images satisfies that agreement. The learning setup must also encourage informative variation. Beginner friendly numerical example: Three different images Imagine three unlabeled photographs : dog A, car B, and tree C. Each image is encoded as a vector of two numbers. Step 1 — Completely collapsed representations. Dog: z A = [1, 1] Car: z B = [1, 1] Tree: z C = [1, 1] The images are different, but all three embeddings are identical. Step 2 — Calculate per coordinate variance. For the first coordinate , the values across these images are [1, 1, 1]. Mean: μ₁ = (1+1+1)/3 = 1. Population variance: Var₁ = [(1−1)²+(1−1)²+(1−1)²]/3 = 0. For the second coordinate , values are also [1, 1, 1]. Mean: μ₂ = 1. Population variance: Var₂ = 0. Both dimensions have zero variance , because the encoder assigns the same values to every image. Step 3 — Calculate cross image cosine similarity. For dog and car: cos(z A,z B) = (1×1+1×1)/(√2×√2) = 2/2 = 1 Every pair of these nonzero embeddings has cosine similarity 1 . An embedding based retriever cannot use them to distinguish the photographs. Step 4 — Contrast with a noncollapsed toy representation. Suppose a different encoder produces: Dog: [1, 0] Car: [0, 1] Tree: [−1, 0] Now the first coordinate [1, 0, −1] has mean 0 and population variance: Var₁ = (1²+0²+(−1)²)/3 = 2/3 ≈ 0.667 The second coordinate [0, 1, 0] has mean 1/3 and population variance: Var₂ = [(0−1/3)²+(1−1/3)²+(0−1/3)²]/3 = 2/9 ≈ 0.222 Also, cos(dog,car) = 0 instead of 1. These illustrative vectors demonstrate diversity, not guaranteed semantic quality: downstream performance still needs evaluation. Numerical example: Dimensional collapse Suppose the encoder is supposed to output three features, but produces: Image A: [−1, 0, 0] Image B: [0, 0, 1] Image C: [1, 0, 0] The coordinate variances are: Dimension 1: Values [−1, 0, 1], mean 0, variance 2/3 . Dimension 2: Values [0, 0, 0], mean 0, variance 0 . Dimension 3: Values [0, 1, 0], mean 1/3, variance 2/9 . These embeddings are not all equal , so there is no complete collapse . But one of three dimensions is entirely unused. In more subtle cases, many coordinates have nonzero variance yet are strongly correlated, leaving a low effective rank : the model uses too few independent feature directions. A single constant coordinate is not automatically a catastrophic failure . Dimensional collapse becomes concerning when the representation's usable capacity or downstream utility is meaningfully reduced. How can you detect representation collapse? Per feature variance or standard deviation: Compute how much each coordinate changes across a batch of different source images . Many near zero values are a warning sign. Pairwise similarity: Check cosine similarity between embeddings from unrelated or different inputs. Almost universally high similarities can indicate near identical directions. Covariance and effective rank: Inspect a covariance matrix's eigenvalues or the embeddings' singular value spectrum. A few dominant values with many near zero values can indicate dimensional collapse even if individual variances appear healthy. Downstream evaluation: Fit a simple linear probe or test image retrieval. Check whether the features actually distinguish useful categories or instances. Use multiple diagnostics. High similarity between genuinely related images may be desirable; cosine similarity is undefined for zero vectors; artificially increasing vector magnitude can raise unnormalized variance without adding useful information. How can collapse be prevented? Contrasting examples: Many contrastive objectives require an anchor to distinguish its matched view from other images. Constant outputs then cannot succeed at discriminating the candidate views. Variance regularization: Methods such as VICReg explicitly penalize feature standard deviations that fall below a chosen threshold across a training batch. This resists all constant solutions without requiring negative pairs. Decorrelating features: Covariance regularization (VICReg) or redundancy reduction (Barlow Twins) encourages different embedding dimensions to carry less redundant information, addressing dimensional underuse. Architectural and optimization choices: Predictor networks, stop gradient operations, normalization, asymmetric branches, and momentum updated targets can help in specific training systems. They are not interchangeable, universally sufficient fixes . Task aware augmentation: Preserve information relevant to the intended downstream task rather than enforcing invariance to everything. The specific no negative learning mechanisms used by BYOL and the teacher–student method DINO are covered in their later curriculum articles; this question establishes the underlying problem and general ways to diagnose it. Why does collapse matter in applications? Classification: If every input gets the same features, even a good linear classifier has no basis for distinguishing classes. Visual search: Nearly identical embeddings make similarity rankings uninformative. Transfer learning: Pretraining can appear to optimize its training loss yet fail to yield reusable features. Training monitoring: Track variance, correlation/effective rank, and downstream performance—not just the training objective. Questions 268 (SimCLR) and 269 (MoCo) discuss two contrastive training frameworks; their complete architectures are not repeated here. Key takeaway: Self supervised features should be similar for related views yet remain informative across different images. Complete collapse removes all variation; dimensional collapse removes too many independent directions. Research: Bardes, Ponce & LeCun, VICReg: Variance Invariance Covariance Regularization for Self Supervised Learning ; Zbontar et al., Barlow Twins: Self Supervised Learning via Redundancy Reduction ; Li, Efros & Pathak, Understanding Collapse in Non Contrastive Siamese Representation Learning .