What is batch normalization?

Batch normalization is a technique that normalizes a layer's activations during training and then applies learned scaling and shifting parameters.

Its goal is to make training more stable and easier to optimize.

For a mini-batch, batch normalization roughly performs:

Normalize activations
      ↓
Apply learned scale (γ)
      ↓
Apply learned shift (β)

Why is batch normalization useful?

It can help by:

• Keeping activation distributions more controlled during training.
• Allowing higher learning rates in some setups.
• Making optimization less sensitive to parameter initialization.
• Improving training stability in many architectures.

Training vs inference

During training, batch normalization uses statistics computed from the current mini-batch.

During inference, it typically uses running averages collected during training instead of statistics from the current input batch.

Training  → batch statistics
Inference → stored running statistics

Does it prevent overfitting?

Batch normalization can sometimes have a mild regularizing effect because batch statistics introduce noise during training, but its primary purpose is training stability and optimization, not regularization.

For related concepts, see “What is batch size in deep learning?”, “What is a learning rate?”, and “What is regularization in deep learning?”.

In short, batch normalization normalizes intermediate activations during training and then learns how to rescale them, helping many neural networks train more reliably.