What is layer normalization?

Layer normalization is a normalization technique that normalizes the activations within each individual example rather than across a mini-batch.

It is commonly used in sequence models and Transformer architectures because its behavior does not depend on the batch size.

How layer normalization works

For a given example, layer normalization computes statistics across the features of that layer:

• Calculate the mean of the activations.
• Calculate their variance.
• Normalize the activations using those values.
• Apply learnable scale and shift parameters.

A simplified form is:

normalized = (x - mean) / sqrt(variance + ε)

Then the normalized values are adjusted using learned parameters:

output = γ × normalized + β

where γ and β are learned during training.

Why is layer normalization useful?

It can help:

• Stabilize the scale of activations.
• Make optimization more consistent.
• Work well even with very small batch sizes.
• Support sequence models where examples may have different lengths.

How is it different from batch normalization?

Batch normalization uses statistics collected across examples in a mini-batch, while layer normalization computes statistics within each example.

That difference makes layer normalization especially useful in architectures such as Transformers.

The detailed comparison is covered separately in “What is the difference between batch normalization and layer normalization?”.

In short, layer normalization normalizes the features of each individual example, helping stabilize training without depending on statistics from other examples in the batch.