What is focal loss in deep learning?

Focal loss is a modified cross-entropy loss that reduces the contribution of easy, correctly classified examples so training pays relatively more attention to hard examples. It was introduced with RetinaNet to address the huge number of easy background examples in dense object detection.

What changes compared with cross-entropy?

For a binary classification example, let pₜ be the model's predicted probability of the correct class. A commonly used focal loss is:

`FL(pₜ) = −αₜ (1 − pₜ)^γ log(pₜ)`

• `−log(pₜ)` is the ordinary cross-entropy term.
• `(1 − pₜ)^γ` is the *focusing factor*: it becomes small when the model already predicts the correct class confidently.
• `γ` (gamma) controls the strength of focusing. γ = 0 removes the focusing factor and recovers α-weighted cross-entropy; larger positive values suppress easy examples more strongly.
• `αₜ` (alpha) is an optional class-dependent balancing weight, separate from gamma. In binary implementations, positive and negative targets may use different weights.

Beginner-friendly numerical example

Suppose an object detector sees one easy background location and one difficult object location. To isolate gamma's effect, set `αₜ = 1` and `γ = 2`:

• Easy example: pₜ = 0.9 means 90% probability for the correct class. Cross-entropy is −ln(0.9) ≈ 0.1054. The focusing factor is (1 − 0.9)² = 0.01, so focal loss ≈ `0.00105`.
• Hard example: pₜ = 0.2 means only 20% probability for the correct class. Cross-entropy is −ln(0.2) ≈ 1.6094. The focusing factor is (1 − 0.2)² = 0.64, so focal loss ≈ `1.0300`.

Under the same gamma, the easy example's loss is reduced by 99% relative to its own cross-entropy loss, while the hard example's loss is reduced by 36%. That is why the hard example contributes much more *relative* training signal. These are per-example losses, not overall model accuracy.

Why is this useful?

• Dense object detection: Thousands of candidate locations or anchors may be background; many become easy negatives. Without focusing, their combined contribution can dominate training.
• Class imbalance: Focal loss can help when many examples are easy, but it does not automatically solve every class-imbalance problem. Adjusting α and sampling can also matter.
• Other applications: It can be evaluated for difficult or imbalanced classification and segmentation tasks, but it is not always better than ordinary cross-entropy.

Practical settings and limitations

• `γ = 2` is a common starting point from RetinaNet, not a universally optimal setting.
• `α = 0.25` and `γ = 2` are familiar RetinaNet/Torchvision defaults; alpha's interpretation depends on the implementation and class labeling.
• Very large gamma can down-weight useful examples too aggressively. Tune alpha, gamma, and the loss reduction using validation results.
• Focal loss is an optimization objective during training; it does not change the detector's architecture or guarantee improved performance.

Key takeaway: Cross-entropy penalizes mistakes based on the correct-class probability. Focal loss adds an adjustable factor that greatly lowers easy examples' influence, letting harder examples matter more.

Related: “What is cross-entropy loss?” and “What is class imbalance and how does it affect deep learning models?”

Research: Focal Loss for Dense Object Detection — Lin et al. (ICCV 2017). Implementation: Torchvision sigmoid_focal_loss.