What is momentum in gradient descent?

Momentum is a technique that helps gradient descent move more consistently by combining the current gradient with information from previous updates.

Instead of changing parameters using only the latest gradient, momentum keeps a running update direction, often called velocity.

How it works

A simplified form is:

velocity = momentum × previous_velocity + gradient

parameter = parameter - learning_rate × velocity

The momentum coefficient is often set to a value such as 0.9, meaning recent update directions continue to influence the next step.

Why is momentum useful?

Momentum can help:

• Accelerate movement in directions where gradients consistently point the same way.
• Reduce oscillation when gradients repeatedly change direction.
• Improve convergence in narrow or curved regions of the loss landscape.

A useful intuition is a ball rolling downhill: once it builds speed in a useful direction, it tends to keep moving instead of reacting completely to every small change in slope.

Momentum does not replace the learning rate; both work together to determine how parameters are updated.

For the underlying optimization method, see “What is gradient descent?”. For step size, see “What is a learning rate?”.

In short, momentum makes gradient descent remember part of its previous update direction, which can reduce oscillation and help optimization move faster toward a good solution.