How does momentum accelerate neural network training?

Momentum accelerates neural network training by combining the current gradient with a running average of previous update directions.

This gives the optimizer a kind of velocity, so it can move faster in directions where gradients consistently point the same way.

Without momentum

Standard gradient descent updates parameters using only the current gradient:

w_new = w_old - learning_rate × gradient

If the loss surface is narrow or curved, updates may zigzag from side to side and make slow progress toward the minimum.

With momentum

Momentum keeps part of the previous update:

v_t = β × v_(t-1) + gradient_t

w_new = w_old - learning_rate × v_t

where β controls how much past direction is retained.

A common value is around:

β = 0.9

Why does this accelerate training?

If gradients repeatedly point in a similar direction, momentum accumulates that direction over time.

This can:

• Increase movement along useful directions.
• Reduce side-to-side oscillation.
• Help the optimizer move through shallow regions faster.
• Improve convergence on elongated or uneven loss surfaces.

Simple intuition

Imagine a ball rolling downhill.

Instead of reacting only to the slope directly under it, the ball also carries speed from its previous motion.

That accumulated velocity helps it move faster in a consistent downhill direction.

Does momentum always make updates larger?

No.

If gradients change direction frequently, previous momentum can partially cancel the current direction.

This damping effect is one reason momentum can reduce oscillation.

For the basic definition, see “What is momentum in gradient descent?”.

In short, momentum speeds up training by accumulating consistent gradient directions while smoothing out oscillations, allowing the optimizer to move more efficiently across the loss surface.