What is the exploding gradient problem?

The exploding gradient problem happens when gradients become extremely large as they are propagated backward through a neural network.

When this happens, parameter updates can become too large, making training unstable.

Why does it happen?

During backpropagation, gradients are repeatedly multiplied through many layers.

Gradient at output
      ↓ × large value
larger gradient
      ↓ × large value
very large gradient

If these multiplications keep increasing the gradient magnitude, the values can grow rapidly as they move toward earlier layers.

Why is it a problem?

Exploding gradients can cause:

• Very large parameter updates
• Unstable or oscillating loss
• Numerical overflow or NaN values
• Training that fails to converge

How is it reduced?

Common techniques include:

• Gradient clipping: limits gradient magnitude before the optimizer applies an update.
• Careful weight initialization: helps keep activations and gradients in a reasonable range.
• Appropriate learning rates: smaller updates can improve stability.
• Normalization and architectural choices: can make training more stable in deep networks.

For the related opposite problem, see “What is the vanishing gradient problem?”. For how gradients are computed, see “How does backpropagation work?”.

In short, exploding gradients occur when gradient values grow too large during backpropagation, causing unstable parameter updates and potentially breaking training.