Deep Learning

Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.

How are image patches embedded in a Vision Transformer?

Patch embedding converts small regions of an image into fixed length numerical vectors (tokens) that a Vision Transformer can process. In the original ViT, each non overlapping image patch is flattened and passed through a learned linear projection . How are patch embeddings created? 1. Split: Divide an image of size H × W into non overlapping P × P patches. When both dimensions are divisible by P, the number of patches is N = (H / P) × (W / P). 2. Flatten: A patch with C color channels contains P × P × C numbers. Arrange them into one vector. 3. Project: Apply the same trainable linear layer to every patch, converting its flattened vector to an embedding of size D. Its weight matrix has shape (P × P × C, D). 4. Add position: Add a positional embedding to each patch token so the Transformer can distinguish where its image region came from. The original classification ViT also prepends a separate learned class token. Beginner friendly numerical example Suppose we have one 32 × 32 RGB image , choose 8 × 8 patches , and want 64 features per token . Patch grid: 32 / 8 = 4 patches in each direction, so 4 × 4 = 16 patches. Flattened patch length: 8 × 8 × 3 = 192 pixel/channel values. Learned projection: Map each 192 value patch vector to 64 values using a linear layer. Patch token tensor: (1, 16, 64) — one image, 16 patches, 64 features per patch. With a class token: (1, 17, 64) before entering the Transformer encoder. Positional embeddings are added , so they do not create extra tokens. Equivalent PyTorch implementation nn.Conv2d(in channels=3, out channels=64, kernel size=8, stride=8) Applied to (1, 3, 32, 32), this produces (1, 64, 4, 4). Flatten the spatial dimensions and transpose them to obtain (1, 16, 64). With kernel size equal to stride, this convolution performs the same learnable patch projection as flattening each non overlapping patch and applying a shared linear layer. Why not use raw pixel vectors? The learned projection maps patch pixels into the shared feature dimension expected by the Transformer. It does not, by itself, model relationships between different patches; later attention layers do that. Related: “What is a Vision Transformer (ViT)?” explains the complete architecture. Research: An Image is Worth 16x16 Words — Dosovitskiy et al. (ICLR 2021). Documentation: Hugging Face Vision Transformer.

How does a Vision Transformer differ from a CNN?

A CNN and a Vision Transformer (ViT) both learn visual features, but they combine image information differently. A conventional CNN applies shared convolutional filters to local neighborhoods ; a standard ViT uses self attention to connect image patches, including patches far apart. Key differences Feature extraction: A CNN slides learned filters over nearby pixels or feature map locations. A ViT uses patch representations and attention layers to learn relationships among tokens. Local versus global context: A 3×3 convolution directly examines a small neighborhood; stacking convolution layers grows the receptive field. In a standard ViT encoder, each token can attend to every other token within a single attention layer. Built in assumptions (inductive bias): CNNs explicitly build in locality and filter sharing across positions, which encourages translation equivariant feature extraction. Vanilla ViT has less image specific inductive bias and uses position information to represent spatial layout. How interactions are chosen: Convolution uses learned filters shared across image locations. Self attention computes input dependent weights linking tokens. Training and compute: CNNs can be strong choices with limited training data. The original ViT benefited especially from large scale pretraining; pretrained ViTs and improved training methods can also work well with smaller task specific datasets. Full self attention has token pair computation that grows approximately with the square of the number of tokens. Beginner friendly practical comparison Imagine a 4×4 grid of 16 visual features representing parts of a photograph of a dog. CNN approach: A 3×3 convolution at an interior location directly combines up to 9 nearby positions . More convolution layers can carry information from the dog's tail to its head. ViT approach: Self attention allows the token representing the tail to interact directly with the token representing the head in one encoder layer. With 16 tokens, an attention head computes a 16×16 matrix of 256 token to token scores before normalization. This is an illustration of information flow , not an apples to apples speed comparison: attention and convolution have different arithmetic and memory costs. Which should you choose? Consider a CNN when you want a strong locality aware baseline or need to optimize for limited data, memory, or device specific latency. Consider a pretrained ViT when strong pretrained representations and flexible long range interactions are useful. Benchmark both for your dataset, accuracy target, latency, and hardware. A ViT is not automatically more accurate or efficient: modern CNNs such as ConvNeXt remain competitive. In short: CNNs build visual understanding from local patterns outward ; ViTs use attention across patch tokens . Neither is universally better. Related: “What is a Vision Transformer (ViT)?”, “How are image patches embedded in a Vision Transformer?”, and “What is ConvNeXt?” Research: An Image Is Worth 16x16 Words (ICLR 2021) and A ConvNet for the 2020s (CVPR 2022).

What is a Swin Transformer and why does it use shifted windows?

Swin Transformer (Shifted Window Transformer) is a hierarchical vision Transformer that computes self attention inside small, non overlapping windows rather than across every image patch at once. It shifts the window boundaries between successive blocks so patches in neighboring windows can exchange information. How do the shifted windows work? First block (W MSA): Partition the feature map into windows and calculate attention only among tokens inside each window. Next block (SW MSA): Shift the window partition by about half a window. Tokens that were separated by an earlier window boundary can now attend to one another. Repeat: Alternating ordinary and shifted windows lets information spread beyond a single window over multiple layers. Build a hierarchy: Deeper stages merge neighboring patches, creating lower resolution, richer feature maps useful for object detection and segmentation. Simple numerical example Imagine an 8 × 8 grid of 64 patch tokens , with a 4 × 4 window size . Ordinary windows: The grid becomes 2 × 2 = 4 windows, each containing 4 × 4 = 16 tokens. Local attention scores: Each window computes a 16 × 16 attention score matrix. Across four windows that is 4 × 256 = 1,024 token pair scores per attention head. Global attention comparison: One full attention matrix over 64 tokens would contain 64 × 64 = 4,096 token pair scores per head. Shifted windows: The next block shifts the partition by 2 tokens vertically and horizontally . A token near the old boundary can now interact with tokens from the neighboring old window. These score counts are illustrative; they exclude projections and other computation. Practical implementations use cyclic shifts and attention masks to prevent incorrect wraparound connections. Why not keep windows fixed? Fixed windows are efficient, but neighboring windows would remain isolated within each such attention layer. Shifting connects those regions without paying for global attention at every layer. With a fixed window size, the attention computation scales approximately linearly with the total number of image tokens, rather than quadratically as in global attention. Practical use: For a street scene image, details near the boundary between a car and the road can exchange information across windows. Hierarchical features also make Swin useful for image classification, object detection, and semantic segmentation. Key takeaway: Swin combines local attention for efficiency , shifted windows for cross window communication , and multi scale features for vision tasks. It does not compute global attention over the whole image in each block. Research: Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows — Liu et al. (ICCV 2021).

What is U-Net and why is it used for image segmentation?

U Net is a convolutional neural network (CNN) designed for image segmentation: predicting which pixels belong to a target region. Introduced for biomedical images, its U shaped architecture combines a contracting path (encoder) that recognizes visual patterns with an expanding path (decoder) that builds a detailed segmentation mask. How does U Net work? Encoder (downsampling): Convolutional layers extract features while pooling reduces spatial resolution. This helps the network recognize the broader context around an object. Bottleneck: The deepest layers process a compact, information rich feature map. Decoder (upsampling): Upsampling and convolution progressively restore spatial resolution so the model can predict pixel locations. Connections between matching levels: Features from the encoder are passed to the corresponding decoder stages to support accurate localization. How these U Net skip connections preserve spatial detail is covered in the next question. Prediction layer: A final convolution produces pixel wise output scores, which can become probabilities or class labels. Beginner friendly practical example Imagine identifying individual cell regions in a microscope image (foreground versus background). In a simple padded U Net implementation : Input: One grayscale image of shape (1, 1, 256, 256) — batch size 1, one channel, 256 × 256 pixels. Encoder: Reduce the spatial size, for example 256 × 256 → 128 × 128 → 64 × 64, while learning richer features. Decoder: Upsample, for example 64 × 64 → 128 × 128 → 256 × 256, and predict where cells appear. Output: One foreground score map of shape (1, 1, 256, 256). Pixel example: After sigmoid, a pixel with probability 0.90 can be labeled cell , while one with 0.10 can be labeled background using a threshold of 0.50. This produces a mask showing cell regions , rather than a single label saying that the image contains cells. Output size depends on implementation: the original 2015 U Net used unpadded convolutions and cropped feature maps, so its output was smaller than its input. Why is U Net widely used? Pixel level localization: Useful when object shapes and boundaries matter. Context plus detail: The encoder recognizes what is in the image, and the decoder predicts where it is. Practical applications: Microscopy, medical image segmentation, satellite imagery, and industrial defect segmentation. Key takeaway: U Net is a specific CNN encoder–decoder architecture for segmentation masks . It predicts spatial regions instead of only assigning a class to the whole image. Related: “What is image segmentation in deep learning?” and “What is an encoder decoder architecture?” Research: U Net: Convolutional Networks for Biomedical Image Segmentation — Ronneberger, Fischer, and Brox (MICCAI 2015).

How do U-Net skip connections preserve spatial detail?

U Net skip connections preserve spatial detail by sending high resolution feature maps from the encoder directly to the decoder at matching spatial scales. This helps the decoder draw precise segmentation boundaries while still using the deeper layers' understanding of what the image contains. Why are skip connections needed? Downsampling loses spatial precision: Pooling compresses feature maps, helping recognize objects but making exact pixel boundaries harder to locate. Upsampling alone cannot recover everything: Enlarging a coarse feature map increases its size but cannot reliably recreate fine details that were discarded. Encoder to decoder shortcuts retain detail: U Net passes earlier, higher resolution features to a decoder stage at the corresponding resolution. Concatenate, then refine: Standard U Net joins the encoder and decoder features along the channel dimension and uses subsequent convolutions to combine them. This is different from a ResNet's elementwise addition. Simple numerical example Imagine segmenting the thin boundary of a cell in a microscope image. In a padded U Net variant , consider one decoder stage: Encoder feature map saved earlier: (1, 64, 128, 128) — 64 channels at 128 × 128 resolution. Decoder feature map before upsampling: (1, 128, 64, 64). After upsampling and channel reduction: (1, 64, 128, 128). Skip connection: Concatenate the saved encoder map with the upsampled decoder map: 64 + 64 = 128 channels. Combined tensor: (1, 128, 128, 128); subsequent convolutions can reduce this to (1, 64, 128, 128) and refine the boundary. The encoder features can still indicate precisely where the cell edge appears, while the decoder features provide the broader context that the region is a cell. Together they help make the pixel level mask less coarse. Important details Skip connections transfer learned features , not untouched original pixels. They improve localization but do not guarantee perfect boundaries. In the original 2015 U Net , unpadded convolutions made encoder and decoder maps different sizes, so the encoder features were cropped before concatenation . Same padded implementations often match spatial sizes directly. These skip paths can also aid gradient flow, but their distinctive role in U Net is combining fine spatial detail with semantic context . Key takeaway: Without U Net's high resolution skip features, the decoder may know what object is present but struggle to reconstruct exactly where its edges are . Related: “What is U Net and why is it used for image segmentation?” and “What is a skip connection in a neural network?” Research: U Net: Convolutional Networks for Biomedical Image Segmentation — Ronneberger, Fischer, and Brox (MICCAI 2015).

What is a Feature Pyramid Network (FPN)?

A Feature Pyramid Network (FPN) is a CNN feature extraction architecture that produces strong feature maps at several spatial resolutions, so a detector can recognize objects of different sizes. It combines coarse, semantically rich features from deep layers with finer resolution features from earlier layers. How does an FPN work? Bottom up backbone: A CNN such as ResNet produces a hierarchy of feature maps, often called C2, C3, C4, and C5. Deeper maps are spatially smaller but carry stronger semantic information. Top down pathway: Starting from the deepest feature map, FPN upsamples its features to the resolution of the next shallower level. Lateral connections: A 1×1 convolution projects the matching backbone feature map to a fixed channel count. FPN adds it elementwise to the upsampled top down map, combining fine location information with higher level semantics. Output refinement: A 3×3 convolution smooths each resulting pyramid feature map. The output levels, typically P2 to P5, are used by detection heads at different spatial scales. Beginner friendly numerical example Imagine detecting small traffic signs and large buses in a 256×256 road image. For illustration, a CNN backbone produces: C4: (1, 512, 16, 16) — batch, channels, height, width. C5: (1, 1024, 8, 8) — deeper, coarser features. An FPN with 256 output channels combines them like this: 1. Project C5 with a 1×1 convolution: 1024 → 256 channels, giving (1, 256, 8, 8) for P5 (after its output refinement). 2. Upsample the top down 8×8 feature map to 16×16: (1, 256, 16, 16). 3. Project C4 with a separate 1×1 convolution: 512 → 256 channels, giving (1, 256, 16, 16). 4. Add the upsampled and lateral features elementwise: (1, 256, 16, 16) + (1, 256, 16, 16) produces (1, 256, 16, 16) for P4 after 3×3 refinement. The detector can use a finer pyramid level for small traffic signs and a coarser level for larger buses , instead of relying on one feature map resolution. The exact level assignment depends on the detector. How is this different from U Net? FPN: Builds multiple feature maps (P2–P5) for scale aware predictions; its standard lateral merge uses elementwise addition . U Net: Its decoder progressively reconstructs a dense segmentation output ; the original U Net merges matching encoder features by channel concatenation . Both use top down information and cross level connections, but their standard output structures and merging operations are different. Key takeaway: FPN is useful because it makes semantically meaningful features available at high and low resolutions with modest extra computation. The general reason multi scale fusion helps detection is covered in the next question. Research: Feature Pyramid Networks for Object Detection — Lin et al. (CVPR 2017). Implementation: Torchvision FeaturePyramidNetwork.

How does multi-scale feature fusion improve object detection?

Multi scale feature fusion improves object detection by combining feature maps that preserve fine spatial detail with feature maps that capture broader semantic context. This helps a detector recognize both small and large objects in the same image. Why is one feature map scale not enough? High resolution features preserve edges and locations, which are valuable for small objects, but may contain weaker object level meaning. Low resolution features carry richer semantic context and are useful for larger objects, but downsampling can make tiny objects difficult to distinguish. Fusion brings these complementary signals together so object detection heads can use detail plus context instead of relying on either alone. What happens during fusion? 1. A visual backbone extracts features at several resolutions. 2. Features from different levels are resized to compatible spatial dimensions and their channels are aligned as needed. 3. The aligned features are combined, for example by addition, concatenation, or learned weighted fusion . 4. Detection heads use the enriched feature levels to predict object classes and bounding boxes at different scales. FPN is one way to build such a pyramid. Other designs, such as BiFPN in EfficientDet , let information flow in both directions and learn how strongly to weight different input features. The main idea is complementary information across scales, not one particular pyramid structure. Beginner friendly numerical example Imagine a 256 × 256 road photograph containing a small 16 × 16 traffic sign and a large 128 × 128 bus . At stride 4 , the feature map is 256 ÷ 4 = 64 pixels wide: 64 × 64 . The traffic sign spans about 16 ÷ 4 = 4 feature cells along each side, while the bus spans about 128 ÷ 4 = 32. At stride 32 , the map is only 256 ÷ 32 = 8 pixels wide: 8 × 8 . The sign spans about 16 ÷ 32 = 0.5 feature cell width, while the bus still spans about 128 ÷ 32 = 4 cells. A detector relying only on the 8 × 8 map can easily lose precise evidence of the tiny sign. A finer map can preserve more of its shape, while deeper features supply context that helps identify it as a sign. These cell spans are approximate scale relationships , not discrete object counts or guaranteed detection outcomes; real features also depend on receptive fields, alignment, and training. Benefits and trade offs Improved small object recognition: Fine details remain accessible alongside semantic information. Better coverage of different sizes: Separate enriched scales can support predictions for signs, people, cars, and buses. More useful context: A local pattern can be interpreted with information from deeper features. Extra cost: Resizing, fusion operations, and additional feature maps consume compute and memory. Poorly aligned or noisy features may not improve accuracy. Key distinction: Multi scale feature fusion combines internal feature maps ; it does not necessarily mean running the original image through the detector multiple times at different input sizes. Related: “What is a Feature Pyramid Network (FPN)?” explains one specific architecture for implementing multi scale fusion. Research: Feature Pyramid Networks for Object Detection — Lin et al. (CVPR 2017). EfficientDet: Scalable and Efficient Object Detection — Tan et al. (CVPR 2020).

What is the difference between semantic, instance, and panoptic segmentation?

Semantic, instance, and panoptic segmentation differ in whether they label every pixel, separate individual objects, or do both. Panoptic segmentation combines the other two into one coherent scene description. The three outputs Semantic segmentation — What class is each pixel? Every pixel gets a category such as car, road, or sky. Two different cars share the same car label; they do not receive separate identities. Instance segmentation — Which individual object is this? Each detected countable object gets its own pixel mask and class, such as car 1 and car 2. Conventional instance segmentation focuses on objects and need not label regions such as road and sky. Panoptic segmentation — What class and, where applicable, which object? Every pixel gets a semantic category; pixels belonging to countable objects also get an instance identity. The output covers the whole scene with non overlapping segments . Things versus stuff Things: Countable objects, such as cars, people, and bicycles. Panoptic segmentation distinguishes individual instances of these categories. Stuff: Broad or amorphous regions, such as road, sky, and grass. They receive semantic labels but do not require individual object IDs . Beginner friendly numerical example Suppose a 100 × 100 image has 10,000 pixels showing two cars on a road under the sky. For a simplified, fully partitioned example: Car 1: 600 pixels. Car 2: 400 pixels. Road: 6,000 pixels. Sky: 3,000 pixels. Here is what each task predicts: Semantic: All 1,000 car pixels share the class car; the other 9,000 pixels receive road or sky labels. We cannot tell from this mask alone that there are two cars. Instance: The detector returns separate masks for car 1 ( 600 pixels ) and car 2 ( 400 pixels ). Its instance masks need not cover road or sky. Panoptic: All 10,000 pixels belong to exactly one labeled region: (car, 1) for 600 pixels, (car, 2) for 400, plus road for 6,000 and sky for 3,000. The instance IDs are arbitrary identifiers , not extra classes. Why is panoptic segmentation useful? In a driving scene, a system may need to distinguish each vehicle while also understanding the complete surrounding road and sky . Panoptic segmentation represents both in a single scene level output. Important distinction: Panoptic segmentation labels visible image regions ; it does not automatically recover hidden parts of occluded objects. The proposed Panoptic Quality (PQ) metric evaluates the combined task. Related: “What is image segmentation in deep learning?” covers the general segmentation task. This answer focuses on the different outputs and the added scope of panoptic segmentation. Research: Panoptic Segmentation — Kirillov et al. (CVPR 2019). Panoptic Feature Pyramid Networks — Kirillov et al. (CVPR 2019).

What is Intersection over Union (IoU) in image segmentation?

Intersection over Union (IoU), also called the Jaccard index, measures how closely a predicted segmentation mask overlaps the correct (ground truth) mask. It is widely used to evaluate image segmentation. How is IoU calculated? IoU = Intersection / Union Intersection: Pixels labeled as the target class in both the predicted and ground truth masks. Union: Pixels labeled as the target class in either mask, counting shared pixels only once. In pixel classification terms: IoU = TP / (TP + FP + FN) , where TP = correctly predicted target pixels, FP = predicted target pixels that are actually background, and FN = actual target pixels that the model missed. Beginner friendly numerical example Imagine a model segmenting a tumor region in a medical image: Ground truth tumor mask: 50 pixels. Predicted tumor mask: 60 pixels. Overlapping tumor pixels (TP): 40. False positive pixels (FP): 60 − 40 = 20. False negative pixels (FN): 50 − 40 = 10. Union: 40 + 20 + 10 = 70 pixels, or 50 + 60 − 40 = 70. IoU: 40 / 70 ≈ 0.5714, or 57.14% . An IoU of 57.14% means that about 57% of the pixels in the union of the two tumor masks overlap correctly. It does not mean that 57% of all image pixels were classified correctly. How do you interpret the score? 1.0 (100%): Perfect mask overlap. 0.0 (0%): No overlap, provided the union is not empty. Higher IoU: Generally indicates closer agreement between the predicted and ground truth regions. True negative background pixels are excluded from the target class IoU formula. This matters when the target occupies only a small part of the image. What about multiple classes? For semantic segmentation with classes such as road, car, and sky, calculate an IoU separately for each class , then average the selected class scores to obtain mean IoU (mIoU) . Clearly state whether background or absent classes are included; toolkits differ in how they handle a class with zero union (absent from both prediction and ground truth). Important: For segmentation, IoU compares sets of mask pixels . In object detection, the same idea can compare bounding boxes , but that is a different application. Key takeaway: IoU directly measures how much of the predicted region and true region overlap, relative to the total region covered by either mask. Related: “What is image segmentation in deep learning?” The Dice coefficient is a related overlap metric discussed separately. References: Keras IoU and MeanIoU documentation and TorchMetrics Jaccard Index documentation.

What is the Dice coefficient in image segmentation?

The Dice coefficient (Dice similarity score) measures how closely a predicted segmentation mask matches the ground truth mask. It is an overlap metric commonly used in medical image segmentation, especially when the target region is small compared with the background. Dice coefficient formula Dice = (2 × overlapping pixels) / (predicted target pixels + ground truth target pixels) Equivalently, Dice = 2TP / (2TP + FP + FN) , where: TP (true positives): Target pixels correctly included in the predicted mask. FP (false positives): Background pixels incorrectly predicted as target. FN (false negatives): Target pixels missed by the prediction. The coefficient ranges from 0 to 1 for non empty masks: 1 means perfect overlap; 0 means no overlap. Beginner friendly numerical example Imagine a model highlighting a leaf in a photograph: Ground truth leaf region: 80 pixels. Predicted leaf region: 70 pixels. Correctly overlapping pixels (TP): 60. Extra predicted pixels (FP): 70 − 60 = 10. Missed leaf pixels (FN): 80 − 60 = 20. Dice = (2 × 60) / (80 + 70) = 120 / 150 = 0.80 So the model's Dice coefficient is 0.80 (80%) for the leaf mask. This is a score of agreement between the two regions , not 80% pixel accuracy over the entire photograph. Why use Dice for segmentation? Focus on the target region: Correct background pixels (true negatives) do not appear in the formula. This is helpful when most pixels are background. Easy to interpret: Higher scores indicate better overlap between predicted and reference masks. Equivalent to pixel level F1: For a binary target mask, the Dice coefficient equals the F1 score calculated from TP, FP, and FN. How does Dice relate to IoU? Both evaluate overlap, but use different denominators. For the same non empty masks , Dice = 2 × IoU / (1 + IoU) . Their values differ, so a Dice score should not be reported as an IoU score. The IoU calculation is covered in the previous question. Practical caveats: For multiclass segmentation, calculate Dice per class and specify how scores are averaged and whether background is included. If both the predicted and ground truth masks are empty, the basic formula gives 0 / 0 ; libraries may ignore that case or apply a defined convention. State your evaluation policy. Key takeaway: Dice tells you how strongly two segmentation regions agree by comparing twice their overlap with their combined sizes. Related: “What is Intersection over Union (IoU) in image segmentation?” The following question explains Dice loss versus cross entropy loss for training. References: MathWorks — Dice similarity coefficient, MONAI — DiceMetric, and Metrics for evaluating 3D medical image segmentation (BMC Medical Imaging, 2015).