How does batch normalization work?

Batch normalization works by normalizing a layer's activations using statistics from the current mini-batch, then applying learnable scale and shift parameters.

This helps keep activations in a more controlled range and often makes optimization more stable.

Step 1: Compute the batch statistics

For each feature or channel, batch normalization calculates the mini-batch mean:

μ_B = mean(x)

and variance:

σ_B² = variance(x)

Step 2: Normalize the activations

Each activation is normalized using:

x̂ = (x - μ_B) / sqrt(σ_B² + ε)

The small value ε prevents division by zero.

After this step, the normalized activations have approximately zero mean and unit variance within the mini-batch.

Step 3: Apply learnable scale and shift

Batch normalization does not force every layer to stay permanently standardized.

It learns two parameters:

• γ to scale the normalized value.
• β to shift it.

The final output is:

y = γx̂ + β

This allows the network to learn the most useful activation distribution for the task.

What happens during training?

During training, batch statistics are calculated from the current mini-batch.

The layer also keeps running estimates of the mean and variance for later use during inference.

Why can this help training?

Batch normalization often:

• Makes optimization more stable.
• Allows larger learning rates in many settings.
• Reduces sensitivity to some initialization choices.
• Improves gradient flow in deep networks.

It was originally motivated by internal covariate shift, although that is not considered the complete explanation for its benefits.

For that concept, see “What is internal covariate shift?”.

The next articles cover practical details such as “Where should batch normalization be placed in a neural network?” and “What happens to batch normalization during inference?”.

In short, batch normalization standardizes activations using mini-batch statistics, then restores flexibility with learnable scale and shift parameters.