Deep Learning
Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.
What is internal covariate shift?
Internal covariate shift refers to changes in the distribution of a layer's inputs or activations during training as the parameters of earlier layers are updated. The term was introduced in the original Batch Normalization work to describe one challenge of training deep networks. Why can this happen? Consider two consecutive layers: Layer 1 → Layer 2 When Layer 1 updates its weights, the values it sends to Layer 2 can change. So even if Layer 2's own parameters have not changed yet, the distribution of the inputs it receives may be different from the previous training step. Why was this considered a problem? If a layer constantly receives changing input distributions, optimization may become harder because the layer has to keep adapting to a moving target. This was one motivation for techniques such as batch normalization. How does batch normalization relate to it? Batch normalization normalizes intermediate activations using batch statistics and then applies learnable scale and shift parameters. This can make training more stable and allow larger learning rates. However, modern research suggests that the benefits of batch normalization cannot be explained only by reducing internal covariate shift. Other effects, such as improving the smoothness and conditioning of optimization, also appear to be important. For the normalization technique itself, see “What is batch normalization?” . The next article explains the mechanism in more detail: “How does batch normalization work?” . In short, internal covariate shift describes changing activation distributions inside a network during training, although it is not considered the complete explanation for why batch normalization works.
How does batch normalization work?
Batch normalization works by normalizing a layer's activations using statistics from the current mini batch, then applying learnable scale and shift parameters. This helps keep activations in a more controlled range and often makes optimization more stable. Step 1: Compute the batch statistics For each feature or channel, batch normalization calculates the mini batch mean: μ B = mean(x) and variance: σ B² = variance(x) Step 2: Normalize the activations Each activation is normalized using: x̂ = (x μ B) / sqrt(σ B² + ε) The small value ε prevents division by zero. After this step, the normalized activations have approximately zero mean and unit variance within the mini batch. Step 3: Apply learnable scale and shift Batch normalization does not force every layer to stay permanently standardized. It learns two parameters: γ to scale the normalized value. β to shift it. The final output is: y = γx̂ + β This allows the network to learn the most useful activation distribution for the task. What happens during training? During training, batch statistics are calculated from the current mini batch. The layer also keeps running estimates of the mean and variance for later use during inference. Why can this help training? Batch normalization often: Makes optimization more stable. Allows larger learning rates in many settings. Reduces sensitivity to some initialization choices. Improves gradient flow in deep networks. It was originally motivated by internal covariate shift , although that is not considered the complete explanation for its benefits. For that concept, see “What is internal covariate shift?” . The next articles cover practical details such as “Where should batch normalization be placed in a neural network?” and “What happens to batch normalization during inference?” . In short, batch normalization standardizes activations using mini batch statistics, then restores flexibility with learnable scale and shift parameters.
Where should batch normalization be placed in a neural network?
Batch normalization is commonly placed after a linear or convolutional layer and before the activation function. A typical pattern is: Linear/Conv → Batch Normalization → Activation For example: Conv → BatchNorm → ReLU Why place it before the activation? The linear or convolutional layer produces raw activations. Batch normalization then normalizes those values before the nonlinear activation is applied. This can help keep the inputs to the activation function in a more stable range during training. Is this placement always required? No. Some architectures use different ordering. For example, pre activation residual networks may use: BatchNorm → ReLU → Conv So batch normalization placement should follow the design of the architecture rather than being treated as a universal rule. What about fully connected networks? A common pattern is: Linear → BatchNorm → ReLU The same general idea applies: normalize the layer output before passing it to the activation. What about convolutional networks? A common pattern is: Conv → BatchNorm → ReLU Batch normalization typically normalizes each channel using statistics calculated across the mini batch and spatial dimensions. Should batch normalization be placed after every layer? Not necessarily. It is usually added where it improves optimization and training stability. Some layers or architectures may not need it, and modern architectures sometimes use other normalization methods instead. For the mechanism itself, see “How does batch normalization work?” . In short, the most common placement is after a linear or convolutional operation and before the activation function, but the exact ordering can vary depending on the architecture.
What happens to batch normalization during inference?
During inference, batch normalization stops using statistics from the current mini batch and instead uses the running mean and running variance learned during training. This makes predictions stable and independent of the other examples being processed at the same time. What happens during training? During training, batch normalization calculates the mean and variance from each mini batch. It also keeps running estimates of those statistics across training. These running values summarize the activation distribution seen during training. What changes during inference? At inference time, the model uses: The stored running mean. The stored running variance. The learned scale parameter γ. The learned shift parameter β. The normalization is therefore based on fixed statistics rather than the current batch. Why not use the inference batch statistics? An inference batch may be: Very small. A single example. Different in composition from one request to another. Using its own statistics could make the prediction depend on which other examples happen to be in the same batch. Running statistics avoid that problem. Simplified inference formula The normalized value is approximately: x̂ = (x running mean) / sqrt(running variance + ε) followed by: y = γx̂ + β Why is train/eval mode important? Deep learning frameworks normally switch batch normalization behavior when the model changes from training mode to evaluation mode. If the model is accidentally left in training mode during inference, it may continue using current batch statistics and updating the running estimates, which can produce inconsistent predictions. For the mechanism itself, see “How does batch normalization work?” . The next article explains the stored statistics directly: “What are running mean and running variance in batch normalization?” . In short, batch normalization uses fixed running statistics during inference instead of current batch statistics, ensuring more stable and consistent predictions.
What are running mean and running variance in batch normalization?
Running mean and running variance are moving estimates of the activation mean and variance that batch normalization collects during training and later uses during inference. Why are they needed? During training, batch normalization can calculate statistics from each mini batch. During inference, however, the model may receive only one example or a very small batch. Using that inference batch to calculate normalization statistics could make predictions unstable. Running statistics provide fixed values instead. How are they updated during training? After each mini batch, batch normalization updates its stored estimates using the current batch statistics. A simplified moving average update looks like: running mean = momentum × running mean + (1 momentum) × batch mean running variance = momentum × running variance + (1 momentum) × batch variance The exact convention for the momentum parameter can vary by framework. What does the running mean represent? The running mean estimates the typical mean activation seen for a feature or channel across training batches. What does the running variance represent? The running variance estimates how much those activations typically vary around the mean. How are they used during inference? During inference, normalization uses the stored values: x̂ = (x running mean) / sqrt(running variance + ε) Then the learned scale and shift parameters are applied: y = γx̂ + β Are running statistics trainable parameters? No. They are updated statistics, not parameters learned through gradient descent. The trainable batch normalization parameters are usually γ and β. For the inference behavior, see “What happens to batch normalization during inference?” . In short, running mean and running variance summarize the activation statistics observed during training so batch normalization can normalize inputs consistently during inference.
What is instance normalization?
Instance normalization is a normalization technique that normalizes each individual sample independently, usually across the spatial dimensions of each channel. It is commonly used in image generation and style transfer models. How does it differ from batch normalization? Batch normalization calculates statistics using multiple examples from the mini batch. Instance normalization calculates its statistics separately for each individual example. For an image tensor, it typically computes the mean and variance for each channel across that image's height and width. How does it work? For one sample and one channel, it calculates: mean = average of spatial activations variance = variance of spatial activations Then it normalizes: x̂ = (x mean) / sqrt(variance + ε) and usually applies learnable scale and shift parameters: y = γx̂ + β Why normalize each instance separately? Different images can have very different contrast, brightness, or style related statistics. Normalizing each image independently reduces the influence of those instance specific statistics. This can be useful when the model should focus more on structural content than on absolute appearance statistics. Where is instance normalization commonly used? It is especially common in: Neural style transfer. Image to image translation. Some generative models. Does it depend on batch size? No. Because every sample is normalized independently, instance normalization works consistently even when the batch size is very small or equal to one. What is the trade off? Removing instance specific mean and variance can be useful for style related tasks, but those statistics may contain important information in other problems. So instance normalization is not automatically better than batch normalization. For related methods, see “What is batch normalization?” and the next article “What is group normalization?” . In short, instance normalization normalizes each sample independently, typically per channel across spatial dimensions, making it especially useful for style transfer and image generation tasks.
What is group normalization?
Group normalization is a normalization technique that divides a layer's channels into groups and normalizes the activations within each group. Unlike batch normalization, it does not depend on statistics collected across different examples in the mini batch. How does it work? Suppose a convolutional layer produces: 32 channels If group normalization uses: 8 groups then each group contains: 32 ÷ 8 = 4 channels For each sample, the mean and variance are calculated across the channels inside a group and their spatial positions. The activations are then normalized: x̂ = (x mean group) / sqrt(variance group + ε) and learnable scale and shift parameters are applied afterward. Why is group normalization useful? Because its statistics are calculated within each individual sample, its behavior does not depend strongly on batch size. This makes it especially useful when: GPU memory forces training with very small batches. Batch normalization statistics would be noisy or unreliable. Large image models require high memory per example. How is it different from instance normalization? Instance normalization effectively normalizes each channel separately. Group normalization combines several channels into each normalization group. So it provides a middle ground between normalizing channels individually and normalizing across the whole batch. How is it different from layer normalization? Layer normalization generally normalizes across all selected features within an individual sample. Group normalization instead divides channels into smaller groups and normalizes each group independently. Does group normalization behave differently during inference? Unlike batch normalization, it does not require running mean or running variance. It calculates its statistics from the current sample in both training and inference. For related methods, see “What is instance normalization?” and “What is layer normalization?” . The later article “When should group normalization be used instead of batch normalization?” covers when this method is preferable. In short, group normalization normalizes groups of channels within each individual sample, making it stable even when the training batch size is very small.
What is weight normalization?
Weight normalization is a technique that reparameterizes a neural network's weight vector by separating its magnitude from its direction . Instead of optimizing a weight vector w directly, it is represented as: w = g × v / ||v|| where: v controls the direction of the weight vector. g controls its magnitude. ||v|| is the norm of v. Both g and v are learned during training. Why separate magnitude and direction? When a network learns w directly, changes to its direction and scale are coupled together. Weight normalization separates them. This can make the optimization problem better conditioned and allow gradient based optimizers to update the parameters more smoothly. Conceptually: Direct parameterization: w Weight normalization: direction v + scale g → w Simple example Suppose a neuron has the weight vector: w = [3, 4] Its magnitude is: ||w|| = 5 The direction is therefore: [3/5, 4/5] Weight normalization represents this idea explicitly: A normalized direction vector A separately learned magnitude The model can then change how strongly the neuron responds without necessarily changing the direction of the weight vector. How is it different from Batch Normalization? Weight normalization operates on the model parameters . It does not calculate the mean or variance of activations from a mini batch. That means it does not depend on batch statistics. Batch Normalization: Normalizes activations using batch statistics. Behaves differently during training and inference. Maintains running mean and running variance. Weight Normalization: Reparameterizes weights. Does not require running statistics. Does not depend on the current batch size. So weight normalization and batch normalization solve related optimization problems in very different ways. Does weight normalization normalize activations? No. This is an important distinction. Weight normalization constrains how the weights are parameterized, but the output activations can still have different means and variances. It should not be treated as a direct replacement for every form of activation normalization. Advantages Weight normalization can: Improve optimization in some neural networks. Decouple weight scale from weight direction. Avoid dependence on mini batch statistics. Work consistently with small or variable batch sizes. Limitations It does not automatically provide all the benefits associated with Batch Normalization or Layer Normalization. Its usefulness depends on the architecture and task, and modern networks often use other normalization strategies more frequently. It can also add a small amount of parameterization and implementation complexity. Interview takeaway Weight normalization reparameterizes a weight vector into a learned magnitude and a normalized direction: w = g × v / ||v||. Unlike Batch Normalization, it acts on weights rather than activations and does not depend on batch statistics. Its purpose is mainly to improve the optimization geometry by separating weight scale from direction.
Why is normalization important in deep neural networks?
Normalization is important in deep neural networks because it keeps activations in a more controlled range, which can make training more stable and easier to optimize. Deep networks contain many layers, so unstable activation scales can make optimization difficult as signals move forward and gradients move backward. What problems can normalization help with? Normalization can help: Stabilize activation distributions. Improve gradient flow. Reduce sensitivity to parameter initialization. Make training less sensitive to learning rate choices. Allow deeper networks to train more reliably. Why does scale matter? If activations become extremely large or very small as they pass through many layers, optimization can become unstable. This may contribute to problems such as saturated activation functions or poorly scaled gradients. Normalization keeps the values entering later layers within a more manageable range. Does normalization always use batch statistics? No. Different methods normalize different dimensions: Batch normalization uses statistics involving the mini batch. Layer normalization normalizes features within each sample. Instance normalization normalizes each sample and channel independently across spatial dimensions. Group normalization divides channels into groups and normalizes each group. The right method depends on the architecture and task. Does normalization replace good optimization? No. Normalization can improve training behavior, but model architecture, weight initialization, optimizer settings, learning rate, and data quality still matter. For the individual methods, see “What is batch normalization?” , “What is layer normalization?” , “What is instance normalization?” , and “What is group normalization?” . In short, normalization helps keep internal representations well scaled, making deep neural networks easier and more stable to train.
When should group normalization be used instead of batch normalization?
Group normalization is usually preferred over batch normalization when the training batch size is very small or varies significantly. Batch normalization depends on statistics calculated across the mini batch. When the batch contains only a few examples, those statistics can become noisy and less reliable. When is group normalization a better choice? Group normalization is especially useful when: GPU memory limits training to very small batches. Each example is large, such as high resolution images or video frames. Batch size changes between training runs. Stable normalization is needed regardless of how many examples are in the batch. Why does group normalization handle small batches better? Group normalization calculates statistics within each individual sample. It divides the channels into groups and normalizes each group independently. Because it does not rely on other samples in the batch, its behavior remains similar whether the batch size is: 1 4 32 When might batch normalization still be preferable? Batch normalization often works very well when sufficiently large and representative mini batches are available. It is also widely used in convolutional networks because it can provide strong optimization benefits with relatively little computational overhead. So group normalization is not automatically better; it solves a specific weakness of batch dependent normalization. Typical example In tasks such as object detection or image segmentation, high resolution inputs can consume a large amount of GPU memory. This may force the batch size down to only a few images. In that situation, group normalization can be more reliable than batch normalization. For the underlying method, see “What is group normalization?” . For batch based normalization, see “What is batch normalization?” . In short, use group normalization when small or unstable batch sizes make batch normalization statistics unreliable.