What is the difference between batch normalization and layer normalization?

Batch normalization and layer normalization both normalize activations to make neural network training more stable, but they normalize across different dimensions.

Batch normalization

Batch normalization computes statistics using multiple examples in the same mini-batch.

For each feature or channel, it typically:

• Computes the mean across the batch.
• Computes the variance across the batch.
• Normalizes the activations.
• Applies learned scale and shift parameters.

Because it depends on batch statistics, its behavior can be affected by very small batch sizes.

During inference, batch normalization usually uses running estimates of the mean and variance collected during training.

See “What is batch normalization?” for the dedicated explanation.

Layer normalization

Layer normalization computes statistics within each individual example rather than across the batch.

For a given example, it normalizes across the features of the layer and then applies learned scale and shift parameters.

This means layer normalization does not depend on other examples in the mini-batch.

It therefore behaves consistently even when batch sizes are small or vary between training runs.

See “What is layer normalization?” for the dedicated explanation.

Main differences

• Batch normalization: uses statistics across examples in a mini-batch.
• Layer normalization: uses statistics across features within each individual example.
• Batch normalization: can be sensitive to batch size.
• Layer normalization: is largely independent of batch size.
• Batch normalization: commonly uses running statistics during inference.
• Layer normalization: normally uses the current example’s own statistics during both training and inference.

Where are they commonly used?

Batch normalization is widely used in many convolutional neural networks.

Layer normalization is widely used in sequence models and Transformer architectures.

Neither method is universally better. The appropriate choice depends on the model architecture, batch size, and training setup.

In short, batch normalization normalizes using statistics across a batch, while layer normalization normalizes features within each individual example.