Deep Learning
Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.
How does Dice loss differ from cross-entropy loss?
Cross entropy loss checks how accurate each pixel's predicted class probability is; Dice loss checks how much the predicted foreground region overlaps the correct region. Both can train segmentation networks, but they emphasize different errors. How do they differ? Binary cross entropy (BCE): Penalizes incorrect probabilities pixel by pixel , then usually averages the penalties. It strongly penalizes confident mistakes. Soft Dice loss: Calculates a differentiable approximation of foreground overlap using predicted probabilities , not thresholded 0/1 masks. A common form is 1 − (2 × Σ(pᵢyᵢ)) / (Σpᵢ + Σyᵢ). Class imbalance: If most pixels are background, ordinary averaged BCE may be dominated by easy background predictions. Foreground only Dice loss focuses on the target region overlap. Weighted BCE can also address imbalance. Here pᵢ is the predicted foreground probability and yᵢ is its target label (0 or 1). Implementations often add a small smoothing constant to stabilize Dice when the target or prediction is empty. Beginner friendly numerical example Imagine a 10 pixel segmentation image containing one foreground pixel and nine background pixels . The model assigns the true foreground pixel probability 0.8 . It assigns each of the nine background pixels foreground probability 0.1 . BCE: −[ln(0.8) + 9 × ln(0.9)] / 10 ≈ 0.1171. Soft foreground Dice: Predicted probability sum = 0.8 + 9 × 0.1 = 1.7; true foreground sum = 1; overlap term = 0.8. Dice loss: 1 − (2 × 0.8) / (1.7 + 1) = 1 − 1.6 / 2.7 ≈ 0.4074. Why are these different? BCE averages the ten individual probability penalties, including nine fairly confident background predictions. Soft Dice looks at the entire foreground prediction and penalizes the combined unwanted foreground probability across those nine background pixels. The loss values use different scales and must not be compared as if they were the same metric. Which one should you use? BCE / cross entropy: A useful baseline when learning well calibrated per pixel class probabilities. Use BCEWithLogitsLoss for a binary segmentation model that outputs logits; this applies sigmoid internally in a numerically stable way. Dice loss: Often helpful when the foreground is small or overlap is a central training goal. Combined loss: Many segmentation pipelines use Loss = λ₁ × BCE + λ₂ × Dice loss to benefit from both pixel level supervision and overlap focused training. Weights should be tuned on validation data. For multiclass segmentation , standard cross entropy uses class logits and integer class targets, while Dice may be computed across selected class channels and averaged. Specify whether background and absent classes are included. Key takeaway: Cross entropy asks 'How wrong is each pixel's probability?' Dice loss asks 'How well does the whole predicted region overlap the target?' Related: “What is cross entropy loss?”, “What is class imbalance and how does it affect deep learning models?”, and “What is the Dice coefficient in image segmentation?” Research: V Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation — Milletari et al. (2016); Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations — Sudre et al. (2017). Implementation: MONAI DiceLoss and DiceCELoss.
What is focal loss in deep learning?
Focal loss is a modified cross entropy loss that reduces the contribution of easy, correctly classified examples so training pays relatively more attention to hard examples. It was introduced with RetinaNet to address the huge number of easy background examples in dense object detection. What changes compared with cross entropy? For a binary classification example, let pₜ be the model's predicted probability of the correct class. A commonly used focal loss is: FL(pₜ) = −αₜ (1 − pₜ)^γ log(pₜ) −log(pₜ) is the ordinary cross entropy term. (1 − pₜ)^γ is the focusing factor : it becomes small when the model already predicts the correct class confidently. γ (gamma) controls the strength of focusing. γ = 0 removes the focusing factor and recovers α weighted cross entropy; larger positive values suppress easy examples more strongly. αₜ (alpha) is an optional class dependent balancing weight, separate from gamma. In binary implementations, positive and negative targets may use different weights. Beginner friendly numerical example Suppose an object detector sees one easy background location and one difficult object location. To isolate gamma's effect, set αₜ = 1 and γ = 2 : Easy example: pₜ = 0.9 means 90% probability for the correct class. Cross entropy is −ln(0.9) ≈ 0.1054. The focusing factor is (1 − 0.9)² = 0.01, so focal loss ≈ 0.00105 . Hard example: pₜ = 0.2 means only 20% probability for the correct class. Cross entropy is −ln(0.2) ≈ 1.6094. The focusing factor is (1 − 0.2)² = 0.64, so focal loss ≈ 1.0300 . Under the same gamma, the easy example's loss is reduced by 99% relative to its own cross entropy loss, while the hard example's loss is reduced by 36% . That is why the hard example contributes much more relative training signal. These are per example losses, not overall model accuracy. Why is this useful? Dense object detection: Thousands of candidate locations or anchors may be background; many become easy negatives. Without focusing, their combined contribution can dominate training. Class imbalance: Focal loss can help when many examples are easy, but it does not automatically solve every class imbalance problem. Adjusting α and sampling can also matter. Other applications: It can be evaluated for difficult or imbalanced classification and segmentation tasks, but it is not always better than ordinary cross entropy. Practical settings and limitations γ = 2 is a common starting point from RetinaNet, not a universally optimal setting. α = 0.25 and γ = 2 are familiar RetinaNet/Torchvision defaults; alpha's interpretation depends on the implementation and class labeling. Very large gamma can down weight useful examples too aggressively. Tune alpha, gamma, and the loss reduction using validation results. Focal loss is an optimization objective during training; it does not change the detector's architecture or guarantee improved performance. Key takeaway: Cross entropy penalizes mistakes based on the correct class probability. Focal loss adds an adjustable factor that greatly lowers easy examples' influence , letting harder examples matter more. Related: “What is cross entropy loss?” and “What is class imbalance and how does it affect deep learning models?” Research: Focal Loss for Dense Object Detection — Lin et al. (ICCV 2017). Implementation: Torchvision sigmoid focal loss.
How do one-stage and two-stage object detectors differ?
One stage and two stage object detectors differ in when they decide where objects might be. A one stage detector predicts object classes and bounding boxes directly from image features. A two stage detector first proposes candidate object regions and then classifies and refines those proposals. How do the two approaches work? One stage — direct prediction: Image → backbone/features → detection head → boxes + class scores. The head predicts candidates at predefined feature locations or learned queries/points depending on the architecture, without a separate region proposal stage. Examples include YOLO, SSD, and RetinaNet . Two stage — propose, then refine: Image → shared features → region proposals → per region classification + box refinement. Faster R CNN uses a Region Proposal Network (RPN) to propose candidate boxes; a second detection head examines these regions. Important distinction: 'One stage' does not mean the network has one neural network layer, and 'two stage' does not require two entirely separate neural networks. They describe the detection pipeline , not the number of convolutional layers or forward passes. Beginner friendly practical comparison Imagine a street image containing 3 cars and 2 pedestrians : One stage approach: A YOLO style model computes image features, then directly predicts candidate boxes, classes, and confidence scores. Post processing keeps suitable detections — ideally 3 car boxes and 2 pedestrian boxes . Two stage approach: A Faster R CNN style model might generate 300 candidate regions in its first stage. Its second stage evaluates these proposals, predicts categories, and refines coordinates. After filtering, it might also return 3 car boxes and 2 pedestrian boxes . The number 300 is illustrative and is a proposal budget used in the Faster R CNN paper, not a fixed requirement or the number of final objects. Both designs may produce many initial candidates and use post processing, such as non maximum suppression. Practical trade offs Latency: One stage systems are often attractive for real time applications because they avoid a separate proposal and region processing stage. Actual speed depends on model size, hardware, input resolution, and implementation. Accuracy: The proposal and refinement design can help with localization and challenging scenes, but two stage is not automatically more accurate ; modern one stage models can also be very accurate. Complexity: One stage pipelines generally have simpler detection flows. Two stage pipelines introduce proposal generation and per region processing. When to use: For live camera analytics, start by benchmarking a suitable one stage model. For accuracy focused analysis, compare one and two stage alternatives on the same dataset rather than choosing by architecture label alone. Key takeaway: One stage = predict detections directly. Two stage = propose candidate regions, then classify and refine them. Measure accuracy and latency on your own workload. Related: “What is object detection in deep learning?” covers the basic task. “What is focal loss in deep learning?” covers a training loss associated with RetinaNet. Region Proposal Networks and RoI operations are covered later in this curriculum. Research: You Only Look Once: Unified, Real Time Object Detection — Redmon et al. (CVPR 2016); Faster R CNN: Towards Real Time Object Detection with Region Proposal Networks — Ren et al. (NeurIPS 2015).
What is an anchor box in object detection?
An anchor box is a predefined reference rectangle used by anchor based object detectors to predict object locations. Instead of guessing every bounding box entirely from scratch, the model starts with anchors of different sizes and shapes and learns how to move and resize them. How do anchor boxes work? Place reference boxes: At selected feature map locations, create several anchors centered at the corresponding image position. Vary size: Small anchors are candidates for small objects; larger anchors can better cover large objects. Vary aspect ratio: A square (1:1), tall (1:2 width:height), or wide (2:1) anchor gives different starting shapes. Predict adjustments: For each anchor, the detector estimates object/class confidence and bounding box offsets. Those offsets shift the box center and adjust its width and height. Train using matches: Anchors are typically assigned positive, negative, or ignored labels according to overlap and other matching rules. Exact thresholds depend on the detector. Beginner friendly numerical example Imagine a feature map location corresponding to image center (100, 80) . Suppose the detector places three reference boxes at that center: Square anchor: 40 × 40 pixels, aspect ratio 1:1. Wide anchor: 80 × 40 pixels, aspect ratio 2:1. Tall anchor: 40 × 80 pixels, aspect ratio 1:2. Assume the wide anchor is selected for a nearby car. Its initial center is (100, 80), width 80, and height 40. In a common Faster R CNN style parameterization, let the model predict: Horizontal offset tₓ = 0.25 and vertical offset tᵧ = −0.10. Log scale width offset t w = ln(1.25) and height offset t h = ln(0.75). Decode the offsets relative to the anchor: New center x: 100 + 0.25 × 80 = 120. New center y: 80 − 0.10 × 40 = 76. New width: 80 × exp(ln(1.25)) = 100 pixels. New height: 40 × exp(ln(0.75)) = 30 pixels. Thus the refined bounding box has center (120, 76) and size 100 × 30 pixels . Its corner coordinates are (70, 61) and (170, 91). These values are illustrative; production implementations may normalize regression targets or use different box encodings. How many anchors can there be? Suppose a feature map has 10 × 10 = 100 positions and uses 3 sizes × 3 aspect ratios = 9 anchors per position . The detector considers 100 × 9 = 900 reference boxes on that map. The original Faster R CNN used nine anchors per sliding window location. Why is this useful? Multiple starting sizes and shapes let an anchor based detector handle varied object dimensions. However, producing many anchors increases matching and prediction work, and poorly chosen shapes can hurt detection. Not every modern detector uses anchors ; anchor free models are covered next. Key takeaway: An anchor is a starting box , not a final prediction. The network scores it and learns offsets that turn it into an object specific bounding box. Research: Faster R CNN: Towards Real Time Object Detection with Region Proposal Networks — Ren et al. and SSD: Single Shot MultiBox Detector — Liu et al..
What is anchor-free object detection?
Anchor free object detection predicts objects without first generating predefined anchor boxes of selected sizes and aspect ratios. Instead, the detector can predict object centers or distances from feature map locations directly to bounding box edges. How does it work? FCOS style point prediction: At a feature map location (x, y), the model predicts whether an object is present and four nonnegative distances: left (l), top (t), right (r), bottom (b) . Convert distances into a box: The corners are (x − l, y − t) and (x + r, y + b) in the same coordinate system. CenterNet style detection: Predict a heatmap of potential object centers and estimate each detected object's width and height (and any center offset used by the implementation). Training: Match points or centers to ground truth objects using model specific assignment rules; there is no need to tune a predefined menu of anchor widths and aspect ratios. Beginner friendly numerical example Suppose an FCOS like detector evaluates an image location (100, 80) and predicts distances to the four sides of a car: l = 30 pixels, t = 20 pixels, r = 50 pixels, b = 25 pixels. Top left: (100 − 30, 80 − 20) = (70, 60). Bottom right: (100 + 50, 80 + 25) = (150, 105). Final box dimensions: width 150 − 70 = 80 pixels; height 105 − 60 = 45 pixels. The prediction is therefore box (70, 60, 150, 105) , with size 80 × 45 pixels . These coordinates are illustrative; actual detectors map feature map locations to image coordinates using their stride. How is this different from anchor based detection? Anchor based: Starts from fixed shape reference rectangles and learns corrections. Anchor size, ratio and matching settings are important design choices. Anchor free: Starts from points or centers and predicts the box geometry directly, eliminating predefined anchor shape configuration. Trade off: Anchor free detection can simplify the pipeline, but still needs sensible target assignment, scale handling, and confidence/post processing choices. Anchor free does not automatically mean faster or more accurate. Important: Anchor free and one stage are not synonyms. FCOS and CenterNet are examples of anchor free, one stage detectors; the two terms describe different architectural choices. Key takeaway: Anchor free methods locate objects using points and predicted geometry , rather than beginning with hand configured anchor shapes. Related: “What is an anchor box in object detection?” explains the predefined box alternative. The next question covers bounding box regression in more depth. Research: FCOS: Fully Convolutional One Stage Object Detection — Tian et al. (ICCV 2019) and Objects as Points (CenterNet) — Zhou et al. (2019).
How does bounding-box regression work in neural object detectors?
Bounding box regression is how an object detector learns to place a rectangle accurately around an object. The model predicts continuous coordinates or coordinate corrections and is trained to minimize the mismatch between its box and the labeled ground truth box. What does the model learn? A box can be represented by its corners (x min, y min, x max, y max) or by its center and size (cx, cy, w, h). Regression targets describe the desired output. Anchor based detectors often use normalized center offsets and logarithmic width/height changes relative to a reference box. Anchor free detectors can instead learn distances from a location to the four sides. The model produces predicted regression values . During training, a loss compares them with targets derived from the labeled object. At inference, the predictions are decoded into image coordinate boxes. Only appropriately matched predictions normally receive positive box regression supervision; background candidates primarily contribute to classification/objectness training. Beginner friendly numerical example Suppose a reference box has center (50, 50) , width 40 , and height 20 . The correct object box has center (58, 46) , width 60 , and height 30 . Using a common Faster R CNN style parameterization: Horizontal target: t x = (58 − 50) / 40 = 0.20. Vertical target: t y = (46 − 50) / 20 = −0.20. Width target: t w = ln(60 / 40) = ln(1.5) ≈ 0.4055. Height target: t h = ln(30 / 20) = ln(1.5) ≈ 0.4055. The training target vector is (0.20, −0.20, 0.4055, 0.4055) . If the network predicts these values, decoding yields center (58, 46) and size 60 × 30 , matching the labeled box. Actual implementations may scale or normalize these targets differently. Which loss functions are used? L1 loss: Penalizes the absolute difference between predicted and target regression values. It is easy to understand but has a sharp point at zero. Smooth L1 (Huber style) loss: Acts approximately quadratically for small errors and linearly for larger ones; widely used for box coordinate or offset regression. Its transition parameter and reduction vary by implementation. IoU based losses: Compare the geometry of the decoded boxes , directly considering their overlap. 1 − IoU can be unhelpful when predicted and target boxes do not overlap. Generalized IoU (GIoU) loss: Adds an enclosing box term that can distinguish non overlapping boxes, supplying geometric information when ordinary IoU is zero. Other variants include DIoU and CIoU. Small loss example If the target horizontal offset is 0.20 but the model predicts 0.30, the absolute error is |0.30 − 0.20| = 0.10 . A simple L1 loss for that coordinate is 0.10; training adjusts the network parameters to reduce the regression errors across supervised examples. Important: Bounding box regression learns where the object is, while classification learns what it is. Detectors typically combine both losses during training. Strong box alignment does not guarantee the predicted class is correct. Key takeaway: Bounding box regression transforms learned numerical targets into precisely located rectangles, then improves those predictions using coordinate based or overlap based losses. Related: “What is an anchor box in object detection?” and “What is anchor free object detection?” cover where the reference for the box prediction comes from; this question focuses on training targets and losses . Research: Fast R CNN — Girshick (ICCV 2015); Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression — Rezatofighi et al. (CVPR 2019).
What is a Region Proposal Network (RPN)?
A Region Proposal Network (RPN) is a small neural network that suggests which parts of an image might contain objects. In Faster R CNN, it shares a backbone's feature maps with the final detector, so the detector does not need a separate, slow proposal algorithm. How are region proposals generated? 1. Extract shared features: A CNN backbone turns the image into a feature map. 2. Score candidate locations: A small sliding convolution processes the feature map. For each reference anchor, an RPN head predicts an objectness score (object versus background) and four box adjustments . 3. Form candidate proposals: Apply the predicted adjustments to obtain bounding boxes in image coordinates; clip boxes to image boundaries and discard invalid or very small candidates as configured. 4. Rank and filter: Sort by objectness and retain a configured number of strong candidates. Apply non maximum suppression (NMS) to reduce highly overlapping proposals, then keep a final proposal budget. 5. Pass to the detector: A second stage head extracts region specific features, assigns the actual object class (for example, car or dog), and refines the box. Important: The RPN predicts objectness , not whether a proposed object is a car, bicycle, or pedestrian. Final classification happens in the next detection stage. Beginner friendly numerical example Imagine a feature map of 10 × 10 = 100 locations , with 3 candidate anchors per location . Initial candidates: 100 × 3 = 300 possible boxes. The RPN predicts 300 objectness scores and 300 sets of four box adjustment values . Illustrative filtering: Keep the 100 highest scoring valid proposals before NMS. Suppose NMS removes 70 because they overlap higher scoring proposals too strongly. 30 proposals remain . The second stage evaluates these 30 candidate regions and might return 2 cars and 1 pedestrian after classification and final filtering. The 100 → 30 filtering counts are illustrative , not fixed Faster R CNN defaults. Real pre NMS and post NMS limits, objectness thresholds, and NMS overlap thresholds depend on the implementation and training/inference mode. The original Faster R CNN demonstrated strong detection with 300 proposals per image . Why is the RPN useful? Fewer regions for expensive processing: The second stage spends computation on promising locations rather than exhaustively classifying many windows. Feature sharing: The proposal network and the final detector reuse full image CNN features. Jointly trainable pipeline: Proposal quality improves through supervised objectness and localization objectives, with implementation specific training strategies. How is it different from the previous concepts? Anchor boxes are possible starting rectangles , and bounding box regression adjusts their coordinates. The RPN is the complete proposal generation module that scores, filters, and sends candidate regions to a second stage detector. Exact NMS mechanics are covered separately later. Key takeaway: An RPN answers 'Where might any object be?' ; the second stage detector answers 'What is it, and exactly where is it?' Research: Faster R CNN: Towards Real Time Object Detection with Region Proposal Networks — Ren et al. (NeurIPS 2015).
What is RoI Align and why is it important?
RoI Align (Region of Interest Align) extracts a fixed size feature map from a proposed image region without rounding its boundaries to integer feature map positions. Introduced in Mask R CNN, it uses bilinear interpolation to preserve spatial alignment—particularly important for predicting accurate instance segmentation masks. Why was RoI Align needed? RoI Pooling divides a proposal into bins but rounds the region boundaries and bin coordinates to the feature grid. That quantization can shift which features correspond to object edges. RoI Align retains fractional coordinates and samples feature values at precise positions using bilinear interpolation, then aggregates samples within each bin. The result is a fixed output size (for example, 7×7 or 14×14) while reducing misalignment between object locations and extracted features. How does bilinear interpolation work? Suppose four neighboring feature map samples have values: Top left: 10; top right: 20. Bottom left: 30; bottom right: 40. We want the value at a sample point 25% across from the left and 50% down from the top (x=0.25, y=0.50, measured within these four neighbors). Interpolate across the top : 10×0.75 + 20×0.25 = 12.5. Interpolate across the bottom : 30×0.75 + 40×0.25 = 32.5. Interpolate vertically : 12.5×0.50 + 32.5×0.50 = 22.5. So RoI Align can sample the continuous position and obtain 22.5 , instead of snapping it to a nearby integer grid coordinate and reading only one neighboring value. How is RoI Align used? 1. Take a proposed rectangle expressed in image coordinates and map it to the backbone feature map coordinate system. 2. Subdivide the rectangle into a fixed number of spatial bins, without quantizing the region or bin boundaries . 3. Sample points in every bin; use bilinear interpolation from the four neighboring feature samples . 4. Aggregate the sampled values in each bin (for example, average or max). The resulting fixed size tensor feeds classification, box, or mask heads. For example, if there are 10 proposals , the backbone has 256 feature channels , and RoI Align outputs 7×7 bins, its feature tensor has shape (10, 256, 7, 7) , regardless of the different proposal sizes. Why is spatial alignment important? A small rounding shift can move predicted edges away from actual object boundaries. Bounding box classification can tolerate some spatial shift, but instance segmentation needs much closer pixel to pixel correspondence . Mask R CNN showed that avoiding coordinate quantization substantially improves mask prediction. Important: RoI Align does not itself propose or classify objects. It extracts aligned features for regions supplied to it . Proposal generation is covered by the previous RPN question. Key takeaway: RoI Pooling rounds region coordinates; RoI Align keeps fractional positions and interpolates features. Preserving this alignment improves precise localization and mask quality. Research: Mask R CNN — He et al. (ICCV 2017). Implementation: Torchvision roi align.
What is non-maximum suppression (NMS) in object detection?
Non maximum suppression (NMS) is a post processing algorithm that removes redundant overlapping detection boxes, keeping high confidence predictions for distinct objects. Without it, a detector may draw several boxes around the same car or person. How does standard (hard) NMS work? 1. Start with predicted boxes and their confidence scores. Optionally discard scores below a chosen confidence threshold. 2. Select the highest scoring box and keep it. 3. Compare overlap: Calculate the Intersection over Union (IoU) between this box and each remaining box. For class aware NMS, compare boxes of the same predicted class . 4. Suppress duplicates: Remove lower scoring boxes whose IoU with the kept box is greater than the chosen threshold. 5. Repeat for the highest scoring remaining box until none remain or the desired output limit is reached. Beginner friendly numerical example Suppose a car detector predicts three boxes in an image, using corner coordinates (x1, y1, x2, y2): Box A: (0, 0, 10, 10), car score 0.95 . Box B: (1, 1, 11, 11), car score 0.85 . Box C: (20, 20, 30, 30), car score 0.70 . Set the NMS IoU threshold to 0.50 . Step 1 — Keep A: It has the highest score (0.95). Step 2 — Compare A and B: A's area = 10 × 10 = 100 square pixels. B's area = 10 × 10 = 100 square pixels. Overlap width = 10 − 1 = 9; overlap height = 10 − 1 = 9. Intersection area = 9 × 9 = 81. Union area = 100 + 100 − 81 = 119. IoU(A, B) = 81 / 119 ≈ 0.681 . Because 0.681 0.50 , suppress B: it is a lower confidence, strongly overlapping prediction. Step 3 — Compare A and C: Their boxes do not overlap, so IoU(A, C) = 0. C survives. Final result: Keep A and C ; remove B . The three initial predictions become two detections . What does the threshold control? Lower IoU threshold (e.g., 0.30): Suppresses more overlapping boxes; may accidentally remove two nearby real objects. Higher IoU threshold (e.g., 0.70): Keeps more boxes; may leave duplicate detections. Class aware NMS: Common multi class implementations suppress boxes within the same class, rather than removing overlapping predictions across unrelated classes. Limitations and alternatives Crowded scenes: When two real objects overlap strongly, hard NMS may discard one of them. Score dependence: NMS keeps the higher scoring candidate even if its coordinates are not the best possible. Soft NMS: Instead of instantly deleting an overlapping box, it lowers that box's score according to the overlap, which can help crowded scenes. Key distinction: An RPN generates and filters object proposals for a two stage detector; NMS is the general duplicate removal operation that can be used during proposal filtering or final detection post processing. Key takeaway: NMS means keep the best scoring box, remove overly overlapping lower scoring boxes, and repeat . Its IoU threshold determines what counts as too much overlap. Research: Soft NMS — Improving Object Detection With One Line of Code (ICCV 2017). Implementation: Torchvision nms.
What is Mask R-CNN and how does it perform instance segmentation?
Mask R CNN is a two stage neural network that detects objects and predicts a separate pixel level mask for each detected instance. It extends Faster R CNN by adding a mask prediction head alongside classification and bounding box regression. How does Mask R CNN work? 1. Shared image features: A CNN backbone (often with a feature pyramid) extracts feature maps from the input image. 2. Candidate objects: The first stage supplies likely object regions to the second stage. The detailed Region Proposal Network mechanism is covered in the earlier RPN concept. 3. Aligned region features: Each candidate is converted into fixed size features so the model can make predictions for that particular region. The exact RoI Align interpolation process is explained separately. 4. Three parallel prediction heads: The second stage outputs class scores, box refinements, and an instance mask. 5. Final instances: Retain appropriate detections and convert each selected mask from its region coordinates into the image's coordinates. What do the three heads predict? Classification head: Predicts the object's category, such as person, dog, or car (including a background category where applicable). Bounding box head: Refines the rectangle around that object. Mask head: Predicts a small binary foreground probability mask for that object's region. In the original class specific design, the network predicts one mask channel per foreground category, and the selected class determines which channel is used. Important distinction: The mask head uses independent per pixel sigmoid probabilities, not a softmax requiring each pixel to choose between all object categories. The class head decides what the object is ; the mask head estimates which pixels belong to that instance . Beginner friendly numerical example Imagine a 512 × 512 street photograph containing two people standing beside each other . Suppose the detector retains 2 person boxes , one for each person. For illustration, assume the model has 3 foreground classes and its mask head produces a 28 × 28 probability grid for each class in each retained region. Per person, the class specific mask output has shape (3, 28, 28) , containing 3 × 28 × 28 = 2,352 probability values. For the 2 person regions, the raw mask outputs together have shape (2, 3, 28, 28) . The predicted class person selects one 28 × 28 mask per region; the two selected masks therefore contain 2 × 784 = 1,568 probabilities. Each selected mask is resized to its detected box. A threshold such as 0.5 converts probabilities into foreground/background pixels, yielding two separate person instances rather than one merged person region. The class count and two detections are illustrative. The original Mask R CNN uses class specific masks of resolution m × m (commonly 28 × 28 for the mask output); modern implementations can differ. Why use Mask R CNN? Instance separation: It can produce distinct masks for two overlapping objects of the same class. Detection plus segmentation: Each output object has a category, box, and pixel level silhouette. Practical uses: Identifying individual cells, counting products on shelves, and segmenting people or vehicles. During training, the original design combines classification, box, and mask losses: L = L cls + L box + L mask . Its mask loss is a binary cross entropy calculated on the ground truth class's mask channel for positive regions. Key takeaway: Faster R CNN predicts what and where ; Mask R CNN adds a third head that predicts the shape of each individual object . Related: “What is a Region Proposal Network (RPN)?”, “What is RoI Align and why is it important?”, and “What is the difference between semantic, instance, and panoptic segmentation?” Research: Mask R CNN — He et al. (ICCV 2017). Official ICCV paper.