What is a Feature Pyramid Network (FPN)?

A Feature Pyramid Network (FPN) is a CNN feature-extraction architecture that produces strong feature maps at several spatial resolutions, so a detector can recognize objects of different sizes. It combines coarse, semantically rich features from deep layers with finer-resolution features from earlier layers.

How does an FPN work?

• Bottom-up backbone: A CNN such as ResNet produces a hierarchy of feature maps, often called C2, C3, C4, and C5. Deeper maps are spatially smaller but carry stronger semantic information.
• Top-down pathway: Starting from the deepest feature map, FPN upsamples its features to the resolution of the next shallower level.
• Lateral connections: A 1×1 convolution projects the matching backbone feature map to a fixed channel count. FPN adds it elementwise to the upsampled top-down map, combining fine location information with higher-level semantics.
• Output refinement: A 3×3 convolution smooths each resulting pyramid feature map. The output levels, typically P2 to P5, are used by detection heads at different spatial scales.

Beginner-friendly numerical example

Imagine detecting small traffic signs and large buses in a 256×256 road image. For illustration, a CNN backbone produces:

• C4: (1, 512, 16, 16) — batch, channels, height, width.
• C5: (1, 1024, 8, 8) — deeper, coarser features.

An FPN with 256 output channels combines them like this:

1. Project C5 with a 1×1 convolution: 1024 → 256 channels, giving (1, 256, 8, 8) for P5 (after its output refinement).
2. Upsample the top-down 8×8 feature map to 16×16: (1, 256, 16, 16).
3. Project C4 with a separate 1×1 convolution: 512 → 256 channels, giving (1, 256, 16, 16).
4. Add the upsampled and lateral features elementwise: (1, 256, 16, 16) + (1, 256, 16, 16) produces (1, 256, 16, 16) for P4 after 3×3 refinement.

The detector can use a finer pyramid level for small traffic signs and a coarser level for larger buses, instead of relying on one feature-map resolution. The exact level assignment depends on the detector.

How is this different from U-Net?

• FPN: Builds multiple feature maps (P2–P5) for scale-aware predictions; its standard lateral merge uses elementwise addition.
• U-Net: Its decoder progressively reconstructs a dense segmentation output; the original U-Net merges matching encoder features by channel concatenation.

Both use top-down information and cross-level connections, but their standard output structures and merging operations are different.

Key takeaway: FPN is useful because it makes semantically meaningful features available at high and low resolutions with modest extra computation. The general reason multi-scale fusion helps detection is covered in the next question.

Research: Feature Pyramid Networks for Object Detection — Lin et al. (CVPR 2017). Implementation: Torchvision FeaturePyramidNetwork.