What is cross-entropy loss?

Cross-entropy loss is a loss function commonly used for classification problems. It measures how well the probabilities predicted by a model match the correct class labels.

A prediction receives a small loss when the model assigns high probability to the correct class, and a large loss when it assigns low probability to the correct class.

Simple intuition

Suppose the correct class is cat.

• If the model predicts P(cat) = 0.95, the loss is small.
• If the model predicts P(cat) = 0.10, the loss is much larger.

This means cross-entropy does not only care whether the final class prediction is correct. It also considers how confident the model is.

Binary cross-entropy

For binary classification, cross-entropy is commonly written as:

Loss = -[y log(p) + (1 - y) log(1 - p)]

Here:

• y is the true label, usually 0 or 1.
• p is the predicted probability of the positive class.

Binary cross-entropy is commonly paired with a sigmoid output.

Multiclass cross-entropy

For multiclass classification, the model usually produces one probability for each class, often using softmax.

If the correct class is c, the loss for one example can be written as:

Loss = -log(p_c)

where p_c is the predicted probability of the correct class.

Why is cross-entropy widely used?

Cross-entropy strongly penalizes confident incorrect predictions and provides a smooth training signal that can be optimized with gradient-based methods.

For the broader idea of loss functions, see “What is a loss function in deep learning?”. For the common output functions, see “What is the sigmoid activation function?” and “What is the softmax function?”.

In short, cross-entropy loss measures how closely predicted class probabilities match the true labels, with larger penalties when the model gives low probability to the correct class.