What is internal covariate shift?

Internal covariate shift refers to changes in the distribution of a layer's inputs or activations during training as the parameters of earlier layers are updated.

The term was introduced in the original Batch Normalization work to describe one challenge of training deep networks.

Why can this happen?

Consider two consecutive layers:

Layer 1 → Layer 2

When Layer 1 updates its weights, the values it sends to Layer 2 can change.

So even if Layer 2's own parameters have not changed yet, the distribution of the inputs it receives may be different from the previous training step.

Why was this considered a problem?

If a layer constantly receives changing input distributions, optimization may become harder because the layer has to keep adapting to a moving target.

This was one motivation for techniques such as batch normalization.

How does batch normalization relate to it?

Batch normalization normalizes intermediate activations using batch statistics and then applies learnable scale and shift parameters.

This can make training more stable and allow larger learning rates.

However, modern research suggests that the benefits of batch normalization cannot be explained only by reducing internal covariate shift.

Other effects, such as improving the smoothness and conditioning of optimization, also appear to be important.

For the normalization technique itself, see “What is batch normalization?”.

The next article explains the mechanism in more detail: “How does batch normalization work?”.

In short, internal covariate shift describes changing activation distributions inside a network during training, although it is not considered the complete explanation for why batch normalization works.