What is the vanishing gradient problem?

The vanishing gradient problem happens when gradients become extremely small as they are propagated backward through a deep neural network.

Because earlier layers receive very small gradients, their weights change very little during training. As a result, those layers may learn slowly or almost stop learning.

Why does it happen?

During backpropagation, gradients are repeatedly multiplied through many layers.

Gradient at output
      ↓ × small value
smaller gradient
      ↓ × small value
much smaller gradient
      ↓
almost zero

If these multiplied values are repeatedly less than 1, the gradient can shrink exponentially as it moves toward earlier layers.

This was especially common in deep networks using saturating activation functions such as sigmoid or tanh.

Why is it a problem?

When gradients become too small:

• Early layers receive almost no useful update signal.
• Training becomes very slow.
• The network may fail to learn useful low-level representations.
• Increasing depth becomes harder to optimize.

How is it reduced?

Common techniques include:

• Using activation functions such as ReLU or related variants.
• Better weight initialization, such as Xavier or He initialization.
• Residual connections that create shorter gradient paths.
• Normalization techniques that improve training stability.

These techniques are covered separately so their details do not need to be repeated here.

For the underlying gradient calculation, see “How does backpropagation work?” and “What is a gradient in deep learning?”.

In short, vanishing gradients occur when the learning signal becomes too small as it travels backward through many layers, making earlier layers difficult to train.