What is Nesterov accelerated gradient?

Nesterov accelerated gradient (NAG) is a momentum-based optimization method that calculates the gradient after first looking ahead in the direction of the current momentum.

It is designed to make momentum more responsive by estimating where the parameters are about to move before computing the next correction.

How is it different from standard momentum?

Standard momentum calculates the gradient at the current parameter position.

Nesterov momentum first makes a temporary look-ahead step:

look_ahead = w - β × v

and then calculates the gradient at that look-ahead position.

A simplified update is:

v_t = β × v_(t-1) + gradient(look_ahead)

w_new = w_old - learning_rate × v_t

Why does the look-ahead help?

Momentum may build up speed in one direction and overshoot a useful region.

Nesterov momentum checks the slope slightly ahead of the current position.

This gives the optimizer an earlier opportunity to correct its direction.

What are the benefits?

Nesterov accelerated gradient can:

• Reduce overshooting.
• Improve responsiveness to changes in the loss surface.
• Preserve the acceleration benefits of momentum.
• Sometimes converge faster or more smoothly than standard momentum.

Simple intuition

Imagine a ball rolling downhill with momentum.

Standard momentum reacts to the slope where the ball currently is.

Nesterov momentum looks slightly ahead in the direction the ball is already moving and adjusts before it gets there.

For the underlying idea, see “What is momentum in gradient descent?” and “How does momentum accelerate neural network training?”.

In short, Nesterov accelerated gradient improves momentum by computing the gradient at a look-ahead position, allowing the optimizer to correct its direction earlier and often move more efficiently.