What is InfoNCE loss in contrastive learning?

InfoNCE (Information Noise-Contrastive Estimation) is a contrastive loss that teaches a neural-network encoder to give a related (positive) view a higher score than competing negative views. It assigns a softmax score to the correct positive and penalizes the encoder when that score is low.

How is InfoNCE calculated?

For one anchor, one positive and K negative candidates, a common form is:

L_InfoNCE = −ln[ exp(s_pos/τ) / (exp(s_pos/τ) + Σ_k exp(s_neg,k/τ)) ]

• `s_pos`: Similarity between the anchor and positive representation, often cosine similarity.
• `s_neg,k`: Similarity between the anchor and its kth negative.
• `τ > 0`: The temperature, which scales similarity scores before exponentiation. We hold it fixed here; its effects are covered in Question 265.
• Numerator: exp(s_pos/τ), the positive's exponentiated score.
• Denominator: The positive's exponentiated score plus the exponentiated scores of all negatives. The positive must be included.
• Outer `−ln`: Takes the negative natural logarithm of the score assigned to the positive, making the loss small when the model ranks the positive confidently.

The anchor itself is excluded from the candidate choices. This formula expresses a single-anchor loss, not necessarily the full training-batch average.

Beginner-friendly step-by-step numerical example

Imagine one anchor image, a positive second view of that same original photo, and two negatives. Suppose the encoder gives these *illustrative* cosine similarities:

• Positive: s_pos = 0.8.
• Negative 1: s_neg,1 = 0.4.
• Negative 2: s_neg,2 = 0.0.
• Temperature: τ = 0.2.

Step 1 — Scale the similarity scores.

• Positive: 0.8 / 0.2 = 4.
• Negative 1: 0.4 / 0.2 = 2.
• Negative 2: 0 / 0.2 = 0.

Step 2 — Take the exponential of each value.

• Positive: exp(4) ≈ 54.5982.
• Negative 1: exp(2) ≈ 7.3891.
• Negative 2: exp(0) = 1.0000.

Step 3 — Add the three exponentials for the denominator.

D = exp(4) + exp(2) + exp(0) ≈ 62.9872

Notice that the numerator's positive term also appears in the denominator.

Step 4 — Calculate the score assigned to the correct positive.

p_positive = exp(4) / D ≈ 0.8668 (about 86.68%).

This is the softmax probability among these three candidate views that the designated positive is the correct match. It is not a probability that the image belongs to a particular object class.

Step 5 — Calculate the InfoNCE loss.

L_InfoNCE = −ln(0.8668) ≈ 0.1429

Answer: For this anchor and these two negatives, the InfoNCE loss is approximately 0.1429. The similarity scores are invented for a numerical illustration, not measured experimental results.

Why do the negative examples matter?

• Competition: Each negative adds a positive term to the denominator. If a negative's similarity rises while the positive similarity stays unchanged, the positive probability falls and the loss increases.
• No negatives: In this simple formula, removing every negative makes numerator equal denominator. Then p_positive = 1 and the loss is 0 for any score: there is no meaningful contrastive training signal.
• False negatives: A “negative” may accidentally depict the same semantic category as the anchor; forcing it away can hurt representation quality. Pair-construction pitfalls are covered in “What are positive and negative pairs in contrastive learning?” (Question 263).

How is it used in real training?

• Across a batch: Compute the loss for multiple anchors and average their individual losses. Often both views of a positive pair serve as anchors.
• Numerically stable calculation: Use log-softmax, cross-entropy or log-sum-exp instead of directly exponentiating large logits, which could overflow.
• Learned representations: Gradients from the loss update the encoder to make the positive more distinguishable from the alternatives.
• Evaluation: A lower contrastive training loss is not automatically better downstream classification or retrieval. Test representations on held-out data.

Benefits: InfoNCE supplies a differentiable learning signal from image-view relationships even without manually supplied object categories. Limitations: Its effectiveness depends on meaningful pair selection; using many negatives can cost memory and compute, and false negatives can distort learning.

For the overall goal of learning visual embeddings, see “What is contrastive representation learning in deep neural networks?” (Question 262). The next article, “How does temperature affect contrastive learning?” (Question 265), explains what changes when τ varies; this article deliberately keeps the temperature fixed.

Key takeaway: InfoNCE uses the positive's exponentiated similarity as a numerator, divides by the sum over positive and negative candidates, and minimizes the negative logarithm of the resulting positive-match score.

Sources: van den Oord et al. (2018), *Representation Learning with Contrastive Predictive Coding*; Chen et al. (ICML 2020), *A Simple Framework for Contrastive Learning of Visual Representations*.