What happens when the learning rate is too high?

When the learning rate is too high, the optimizer can make parameter updates that are too large, causing training to become unstable or fail to converge.

What happens during an update?

A simplified weight update is:

w_new = w_old - learning_rate × gradient

If the learning rate is very large, even a moderate gradient can produce a huge change in the weight.

What problems can this cause?

A learning rate that is too high can lead to:

• Loss values that oscillate instead of decreasing smoothly.
• The optimizer repeatedly overshooting good solutions.
• Sudden spikes in training loss.
• Unstable parameter values.
• Training that diverges completely.
• NaN or infinite values in severe cases.

Simple intuition

Imagine trying to reach the bottom of a valley.

With a reasonable learning rate, each step moves you closer to the bottom.

With a learning rate that is too high, each step may jump from one side of the valley to the other without settling near the minimum.

How can you recognize it?

Common signs include:

• Training loss increases instead of decreasing.
• Loss changes dramatically between updates.
• Accuracy fails to improve.
• Gradients or parameter values become unstable.

How can it be fixed?

Common approaches include:

• Reduce the learning rate.
• Use a learning-rate scheduler.
• Add learning-rate warm-up when appropriate.
• Check whether exploding gradients are also contributing to instability.

For the broader effect of this hyperparameter, see “How does the learning rate affect neural network training?”.

In short, a learning rate that is too high makes optimization steps too large, which can cause overshooting, oscillation, instability, or complete divergence.