What is U-Net and why is it used for image segmentation?

U-Net is a convolutional neural network (CNN) designed for image segmentation: predicting which pixels belong to a target region. Introduced for biomedical images, its U-shaped architecture combines a contracting path (encoder) that recognizes visual patterns with an expanding path (decoder) that builds a detailed segmentation mask.

How does U-Net work?

• Encoder (downsampling): Convolutional layers extract features while pooling reduces spatial resolution. This helps the network recognize the broader context around an object.
• Bottleneck: The deepest layers process a compact, information-rich feature map.
• Decoder (upsampling): Upsampling and convolution progressively restore spatial resolution so the model can predict pixel locations.
• Connections between matching levels: Features from the encoder are passed to the corresponding decoder stages to support accurate localization. *How these U-Net skip connections preserve spatial detail is covered in the next question.*
• Prediction layer: A final convolution produces pixel-wise output scores, which can become probabilities or class labels.

Beginner-friendly practical example

Imagine identifying individual cell regions in a microscope image (foreground versus background). In a simple padded U-Net implementation:

• Input: One grayscale image of shape (1, 1, 256, 256) — batch size 1, one channel, 256 × 256 pixels.
• Encoder: Reduce the spatial size, for example 256 × 256 → 128 × 128 → 64 × 64, while learning richer features.
• Decoder: Upsample, for example 64 × 64 → 128 × 128 → 256 × 256, and predict where cells appear.
• Output: One foreground-score map of shape (1, 1, 256, 256).
• Pixel example: After sigmoid, a pixel with probability 0.90 can be labeled cell, while one with 0.10 can be labeled background using a threshold of 0.50.

This produces a mask showing cell regions, rather than a single label saying that the image contains cells. Output size depends on implementation: the original 2015 U-Net used *unpadded* convolutions and cropped feature maps, so its output was smaller than its input.

Why is U-Net widely used?

• Pixel-level localization: Useful when object shapes and boundaries matter.
• Context plus detail: The encoder recognizes what is in the image, and the decoder predicts where it is.
• Practical applications: Microscopy, medical-image segmentation, satellite imagery, and industrial defect segmentation.

Key takeaway: U-Net is a specific CNN encoder–decoder architecture for segmentation masks. It predicts spatial regions instead of only assigning a class to the whole image.

Related: “What is image segmentation in deep learning?” and “What is an encoder-decoder architecture?”

Research: U-Net: Convolutional Networks for Biomedical Image Segmentation — Ronneberger, Fischer, and Brox (MICCAI 2015).