What is non-maximum suppression (NMS) in object detection?

Non-maximum suppression (NMS) is a post-processing algorithm that removes redundant overlapping detection boxes, keeping high-confidence predictions for distinct objects. Without it, a detector may draw several boxes around the same car or person.

How does standard (hard) NMS work?

1. Start with predicted boxes and their confidence scores. Optionally discard scores below a chosen confidence threshold.
2. Select the highest-scoring box and keep it.
3. Compare overlap: Calculate the Intersection over Union (IoU) between this box and each remaining box. For class-aware NMS, compare boxes of the *same predicted class*.
4. Suppress duplicates: Remove lower-scoring boxes whose IoU with the kept box is greater than the chosen threshold.
5. Repeat for the highest-scoring remaining box until none remain or the desired output limit is reached.

Beginner-friendly numerical example

Suppose a car detector predicts three boxes in an image, using corner coordinates (x1, y1, x2, y2):

• Box A: (0, 0, 10, 10), car score `0.95`.
• Box B: (1, 1, 11, 11), car score `0.85`.
• Box C: (20, 20, 30, 30), car score `0.70`.
• Set the NMS IoU threshold to `0.50`.

Step 1 — Keep A: It has the highest score (0.95).

Step 2 — Compare A and B:

• A's area = 10 × 10 = 100 square pixels.
• B's area = 10 × 10 = 100 square pixels.
• Overlap width = 10 − 1 = 9; overlap height = 10 − 1 = 9.
• Intersection area = 9 × 9 = 81.
• Union area = 100 + 100 − 81 = 119.
• `IoU(A, B) = 81 / 119 ≈ 0.681`.

Because `0.681 > 0.50`, suppress B: it is a lower-confidence, strongly overlapping prediction.

Step 3 — Compare A and C: Their boxes do not overlap, so IoU(A, C) = 0. C survives.

Final result: Keep A and C; remove B. The three initial predictions become two detections.

What does the threshold control?

• Lower IoU threshold (e.g., `0.30`): Suppresses more overlapping boxes; may accidentally remove two nearby real objects.
• Higher IoU threshold (e.g., `0.70`): Keeps more boxes; may leave duplicate detections.
• Class-aware NMS: Common multi-class implementations suppress boxes within the same class, rather than removing overlapping predictions across unrelated classes.

Limitations and alternatives

• Crowded scenes: When two real objects overlap strongly, hard NMS may discard one of them.
• Score dependence: NMS keeps the higher-scoring candidate even if its coordinates are not the best possible.
• Soft-NMS: Instead of instantly deleting an overlapping box, it lowers that box's score according to the overlap, which can help crowded scenes.

Key distinction: An RPN *generates and filters object proposals* for a two-stage detector; NMS is the general duplicate-removal operation that can be used during proposal filtering or final detection post-processing.

Key takeaway: NMS means keep the best-scoring box, remove overly overlapping lower-scoring boxes, and repeat. Its IoU threshold determines what counts as too much overlap.

Research: Soft-NMS — Improving Object Detection With One Line of Code (ICCV 2017). Implementation: Torchvision nms.