Deep Learning
Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.
What is the receptive field of a neuron in a CNN?
The receptive field of a neuron in a convolutional neural network (CNN) is the region of the original input image that can influence that neuron's activation. Think of it as the part of the image a feature map position can see . How does it grow? A 3×3 convolution with stride 1 looks at a 3×3 region of its input. Two successive 3×3 convolutions with stride 1 give a neuron in the second layer a 5×5 receptive field in the original image. Three such convolutions give a 7×7 receptive field . These examples assume ordinary (non dilated) convolutions, stride 1, and an interior output position unaffected by image boundaries. What determines receptive field size? Kernel size: Larger kernels generally cover a wider region. Network depth: Stacking convolutions expands the input region influencing later features. Stride or pooling: Downsampling can make the receptive field grow faster. Dilation: Spacing kernel positions apart enlarges coverage without increasing the number of kernel weights. Why does it matter? Early CNN features can detect local details such as edges. Deeper features need sufficient spatial context to recognize larger structures, such as object parts or whole objects. Important: The receptive field describes the possible input region that can affect an activation, not how strongly every pixel contributes. The distinction is explored next in “What is the difference between theoretical and effective receptive fields in CNNs?” . For related foundations, see “What is a kernel or filter in a CNN?” and “What is stride in a convolutional neural network?” . Further reading: Computing Receptive Fields of Convolutional Neural Networks.
What is the difference between theoretical and effective receptive fields in CNNs?
The theoretical receptive field (TRF) is the full region of an input image that could influence a CNN neuron. The effective receptive field (ERF) is the smaller region whose pixels actually have a meaningful influence on that neuron's output. How are they different? Theoretical receptive field: Determined by network architecture—kernel sizes, strides, dilation, and depth. It describes all input positions connected to an output through computational paths. Effective receptive field: Describes the strength of each input pixel's contribution. It depends on the model's weights and, for nonlinear networks, can depend on the input. Simple example Imagine a feature map neuron with a theoretical receptive field of 15×15 pixels. All 225 positions can potentially affect it, but the strongest influence might be concentrated near the center. The effective receptive field therefore covers a smaller, more influential portion of that 15×15 area. Research on deep CNNs has shown that this influence often forms an approximately Gaussian shaped pattern : central pixels matter more, while influence weakens toward the edges. This is a common finding, not a universal shape for every network and input. How can the effective receptive field be measured? Select an output neuron, compute its gradient with respect to input pixels using backpropagation, and examine the magnitude of those gradients . Larger magnitudes indicate stronger local sensitivity. Averaging across images can reveal the typical influence pattern. Why does the difference matter? A large theoretical receptive field does not guarantee that a network effectively uses context across that whole region. This matters when detecting large objects or understanding broad spatial structure. For the basic definition, see “What is the receptive field of a neuron in a CNN?” . Next: “What is a dilated convolution and when is it useful?” Research: Understanding the Effective Receptive Field in Deep Convolutional Neural Networks. Further reading: Computing Receptive Fields of Convolutional Neural Networks.
What is a dilated convolution and when is it useful?
A dilated convolution (also called atrous convolution) is a convolution that spaces its kernel sampling positions apart. This expands the receptive field without increasing the number of kernel weights. How does it work? A regular 3×3 convolution samples nine nearby positions. With dilation, it samples the same nine positions but farther apart: Dilation rate 1: covers a 3×3 region. Dilation rate 2: spans a 5×5 region. Dilation rate 3: spans a 7×7 region. For a kernel of size k and dilation rate d, the effective kernel span along one dimension is: effective size = k + (k 1) × (d 1) Example: A 3×3 kernel at dilation 2 spans a 5×5 area but still learns just nine spatial weights per input–output channel pair , rather than the 25 weights of a dense 5×5 kernel. When is it useful? Semantic segmentation: Captures larger image context while retaining dense, high resolution feature maps. Multi scale feature extraction: Different dilation rates capture context at different spatial scales, as in DeepLab's atrous spatial pyramid pooling. Temporal modeling: Dilated 1D convolutions can capture longer range patterns in sequences or audio. With stride 1 and suitable padding, dilation enlarges the field of view without downsampling the feature map. What is the limitation? Large dilation rates sample input positions sparsely. Repeated poorly chosen rates can miss local details or create gridding artifacts , so architectures often mix dilation rates or combine dilated and ordinary convolutions. Remember: Dilation controls how far apart kernel positions are sampled ; stride controls how far the kernel moves between output positions . For prerequisites, see “What is the receptive field of a neuron in a CNN?” and “What is stride in a convolutional neural network?” . Research: Multi Scale Context Aggregation by Dilated Convolutions and Rethinking Atrous Convolution for Semantic Image Segmentation.
What is a grouped convolution in a CNN?
A grouped convolution splits a CNN layer's input and output channels into separate groups, then performs convolution independently within each group. Unlike a standard convolution, each output channel uses only the input channels assigned to its group. How does it work? Standard convolution (groups = 1): Every output channel can combine information from every input channel. Grouped convolution (groups = g): Divide the input and output channels into g equal groups. Each group processes only its own input channels, and the resulting output channels are combined into one output tensor. Example: Suppose a 3×3 layer has 16 input channels and 32 output channels with groups = 4: Each group receives 4 input channels and produces 8 output channels . Each group learns its own filters; there are no direct filter connections between different groups in that layer. The result still contains 32 output channels . Why use grouped convolution? Grouped convolution reduces computation and learned parameters when the input/output channel counts and kernel size are held fixed. Ignoring biases , the parameter count is: parameters = out channels × (in channels / groups) × kernel height × kernel width For this example: Ordinary 3×3 convolution: 32 × 16 × 3 × 3 = 4,608 weights. Four group 3×3 convolution: 32 × 4 × 3 × 3 = 1,152 weights — 75% fewer . Grouped convolutions are used in architectures such as ResNeXt to learn multiple channel group transformations efficiently. What is the trade off? Channels in different groups do not interact within the grouped layer. A later 1×1 convolution or channel shuffling operation can help mix information across groups. Important: The groups setting controls channel connectivity ; dilation controls spacing between spatial kernel samples. They are different operations. Special case: When groups = in channels (and output channels are a suitable multiple), grouped convolution becomes depthwise convolution —the next concept in this series. References: PyTorch Conv2d documentation and Aggregated Residual Transformations for Deep Neural Networks (ResNeXt).
What is a depthwise convolution?
A depthwise convolution applies a separate spatial filter to each input channel, instead of combining all channels in one convolution. It is a special case of grouped convolution that reduces the number of weights and arithmetic operations. How does it work? Suppose a feature map has 16 input channels . A depthwise 3×3 convolution with one filter per channel applies 16 independent 3×3 filters . It produces 16 output channels , one from each input channel. The channels are not mixed together by this operation. Compare the number of filter weights (excluding biases) for 16 input and 16 output channels: Standard 3×3 convolution: 16 × 16 × 3 × 3 = 2,304 weights. Depthwise 3×3 convolution: 16 × 3 × 3 = 144 weights. That is 93.75% fewer weights for this layer configuration. Depthwise convolution can also use more than one filter per input channel ( depth multiplier ). How is it implemented? In PyTorch, a typical depthwise layer is: nn.Conv2d(16, 16, kernel size=3, padding=1, groups=16) Setting groups equal to the number of input channels makes each channel use its own filters. When is it useful? Lightweight CNNs: Efficient architectures such as MobileNet use depthwise operations to reduce computation. Mobile and edge devices: Fewer arithmetic operations and weights can help under tight resource budgets. Limitation: Depthwise convolution learns spatial patterns within individual channels but does not combine information across channels. A subsequent 1×1 pointwise convolution is commonly used for channel mixing; together, the operations form a depthwise separable convolution . Actual runtime speed also depends on the hardware and implementation. Key distinction: In a general grouped convolution, each group can contain several input channels; in a depthwise convolution, each group contains exactly one input channel . Related: “What is a grouped convolution in a CNN?” Next: “What is a pointwise convolution?” Research: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.
What is a pointwise convolution?
A pointwise convolution is a 1×1 convolution that combines information across the input channels at each spatial location. Unlike a 3×3 convolution, it does not look at neighboring pixels. How does it work? At one pixel location, the input feature map contains one value from each input channel. A 1×1 filter computes a weighted combination of those values to produce one output channel. Using multiple 1×1 filters produces multiple output channels. The same filters are applied at every spatial location. Example: Suppose a feature map has 16 input channels and we want 32 output channels . Each 1×1 filter has 16 weights , one per input channel. With 32 filters , the layer has 16 × 32 = 512 weights (excluding biases). The output has 32 channels , while its height and width stay unchanged when stride is 1. PyTorch example: nn.Conv2d(16, 32, kernel size=1) Why is it useful? Channel mixing: Combines features from different channels at the same image position. Channel projection: Increases or reduces channel count—for example, from 64 channels to 16 in a bottleneck block. Efficient CNN design: Often follows a depthwise convolution, which processes each channel separately. Pointwise vs. depthwise convolution Depthwise 3×3: Learns spatial patterns within each channel, without mixing channels. Pointwise 1×1: Mixes across channels at each location, without directly expanding spatial context. Together, they form a depthwise separable convolution . Its computational savings compared with a standard spatial convolution are explored in the next question. Related concepts: “What is a depthwise convolution?” and “What is a grouped convolution in a CNN?” Research: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. Documentation: PyTorch Conv2d.
How does depthwise separable convolution reduce computation?
Depthwise separable convolution reduces computation by replacing one expensive spatial convolution with two smaller operations: a depthwise convolution that filters each input channel independently, followed by a 1×1 pointwise convolution that mixes channels. Why is it cheaper? A standard K×K convolution jointly performs spatial filtering and channel mixing for every output channel. Depthwise separable convolution splits those jobs: 1. Depthwise K×K: Apply one spatial filter per input channel. 2. Pointwise 1×1: Combine the resulting channels to produce the desired output channels. Numerical example Assume 32 input channels , 64 output channels , a 3×3 kernel , and one depthwise filter per input channel. Standard convolution: 3 × 3 × 32 × 64 = 18,432 weights. Depthwise step: 3 × 3 × 32 = 288 weights. Pointwise step: 1 × 1 × 32 × 64 = 2,048 weights. Combined: 288 + 2,048 = 2,336 weights. That is 16,096 fewer weights (about 87.3% reduction) , or about 7.9× fewer multiply accumulate terms per output spatial position under matching output dimensions. Biases and other operations are excluded from this comparison. General formula Let K be the square kernel size, Cin the input channels, and Cout the output channels: Standard: K² × Cin × Cout Depthwise separable: (K² × Cin) + (Cin × Cout) Why is it useful? MobileNet uses this pattern to make CNNs lighter for mobile and embedded vision. It can greatly reduce parameters and arithmetic, although actual latency depends on the hardware and implementation , and the factorization can change model accuracy. Related: “What is a depthwise convolution?” and “What is a pointwise convolution?” Research: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. Documentation: PyTorch Conv2d.
What is global average pooling in a CNN?
Global average pooling (GAP) reduces each CNN feature map to one number by averaging all of its spatial values. It is often used near the end of an image classification network to turn feature maps into a compact feature vector. How does it work? A CNN produces a tensor with shape C × H × W, where C is the number of channels. GAP calculates the mean of all H × W values separately for each channel . The result is one value per channel , or C × 1 × 1. No channel mixing occurs. Simple example Suppose one 2×2 feature map contains: [[1, 3], [5, 7]] Global average pooling returns (1 + 3 + 5 + 7) / 4 = 4. If a second channel contains [[2, 4], [6, 8]], it becomes 5. So two 2×2 channels become the vector [4, 5] . For a batched CNN output shaped (1, 64, 7, 7), GAP produces (1, 64, 1, 1), which can be flattened into 64 features. PyTorch: nn.AdaptiveAvgPool2d((1, 1)) Why use GAP? Fewer classifier parameters: It can replace flattening a large feature map into a large fully connected head; GAP itself has no trainable weights. Compact output: It summarizes each channel regardless of the incoming spatial dimensions. Reduced dependence on exact position: Averaging emphasizes whether features occur, rather than precisely where. Trade off: GAP discards spatial layout, making it less suitable when exact object position matters. A small learned classification layer may still follow GAP. Difference from ordinary average pooling: Local average pooling averages small windows; GAP averages the entire spatial map . Related: “What is pooling in a CNN?” and “What is a feature map in a CNN?” Research: Network In Network. Implementation: PyTorch AdaptiveAvgPool2d.
What is a transposed convolution?
A transposed convolution is a learnable operation that can increase the height and width of a CNN feature map. It is commonly used in decoder networks to turn small, coarse feature maps into larger ones. How does it work? A standard convolution with stride greater than 1 often reduces spatial resolution. A transposed convolution with stride greater than 1 can increase spatial resolution while learning how to combine features. Conceptually, it can be understood as spacing input values apart and applying learned convolutional filters; overlapping contributions are added together. Although it is sometimes called deconvolution , it is not the mathematical inverse of convolution and cannot automatically recover information lost during downsampling. Practical example Suppose a CNN decoder receives 16 feature channels of size 4×4 . We want to produce 8 channels of size 8×8 . Use this PyTorch layer: nn.ConvTranspose2d(16, 8, kernel size=4, stride=2, padding=1) Input shape: (1, 16, 4, 4) (batch, channels, height, width). Output shape: (1, 8, 8, 8). For dilation 1 and output padding 0, the spatial size is (4 1) × 2 2 × 1 + 4 = 8. Thus, the layer doubles height and width while learning how to construct the larger feature maps. Where is it useful? Image segmentation: Decoder layers recover a higher resolution prediction from compact image features. Autoencoders: A decoder expands compressed feature maps. Image generators: Learned upsampling turns lower resolution representations into images. Important limitation: Certain kernel/stride combinations can create uneven overlap and checkerboard artifacts. A common alternative is interpolation followed by an ordinary convolution; the next questions compare these approaches. Related: “What is a convolutional layer?” and “What is a feature map in a CNN?” Research: A guide to convolution arithmetic for deep learning. Documentation: PyTorch ConvTranspose2d.
How does transposed convolution differ from interpolation-based upsampling?
Transposed convolution learns how to increase a feature map's spatial resolution. Interpolation based upsampling uses a fixed resizing rule (such as nearest neighbor or bilinear interpolation). A regular convolution can then be added after interpolation to learn how to refine the enlarged features. How do they differ? Transposed convolution: Uses learned filters to produce a larger feature map in one operation. Kernel size, stride, and padding determine its output shape. Interpolation alone: Enlarges a feature map by copying nearby values (nearest neighbor) or computing weighted averages (bilinear). The resize step has no trainable weights and normally preserves the channel count. Interpolation + convolution (resize convolution): First resizes with a fixed rule, then uses a learned convolution to mix and refine features. Practical example Suppose the decoder input has shape (1, 16, 4, 4) : one image, 16 channels, and a 4×4 feature map. Both approaches below produce (1, 8, 8, 8) . Option A — transposed convolution: nn.ConvTranspose2d(16, 8, kernel size=4, stride=2, padding=1) Option B — interpolate, then convolve: nn.Sequential(nn.Upsample(scale factor=2, mode='bilinear', align corners=False), nn.Conv2d(16, 8, kernel size=3, padding=1)) In Option B, interpolation expands 4×4 to 8×8 without changing the 16 channels ; the following convolution learns to produce eight output channels. Which should you use? Transposed convolution: Useful when learned upsampling is desired in a compact decoder layer. Interpolation + convolution: Gives explicit control over resizing and can reduce the risk of checkerboard artifacts caused by uneven overlap in some transposed convolutions. Neither is always more accurate or faster. Compare output quality, memory, and latency for your architecture. Remember: Both can increase resolution, but upsampling does not guarantee recovery of fine details lost earlier in the network . Related: “What is a transposed convolution?” Next: “Why can transposed convolutions create checkerboard artifacts?” References: PyTorch ConvTranspose2d, PyTorch Upsample, and Deconvolution and Checkerboard Artifacts.