How does RMSprop work?

RMSprop works by scaling each parameter's gradient using a moving average of recent squared gradients.

This gives each parameter an adaptive effective learning rate.

Step 1: Calculate the current gradient

After backpropagation, RMSprop receives the gradient:

g_t

This tells the optimizer how the loss changes with respect to the parameter.

Step 2: Track squared gradients

RMSprop maintains an exponential moving average of squared gradients:

v_t = β × v_(t-1) + (1 - β) × g_t²

This value estimates the recent magnitude of the gradients.

Step 3: Scale the update

The parameter update is divided by the square root of that moving average:

w_new = w_old - learning_rate × g_t / (sqrt(v_t) + ε)

where ε is a small value added for numerical stability.

Why square the gradients?

Squaring makes gradient magnitude positive and gives larger gradients more influence on the moving average.

If a parameter has repeatedly large gradients, v_t becomes larger.

That increases the denominator and reduces the effective update size.

If gradients are smaller, the denominator remains smaller and the parameter can receive relatively larger updates.

Why is this useful?

This adaptive scaling can:

• Reduce oscillation in directions with large gradients.
• Allow faster movement in directions with smaller gradients.
• Handle parameters with very different gradient scales.
• Make optimization more stable on uneven loss surfaces.

How is RMSprop different from Adam?

RMSprop mainly tracks squared gradients.

Adam also tracks a moving average of the gradients themselves, adding a momentum-like component.

For the overview, see “What is the RMSprop optimizer?”.

In short, RMSprop tracks recent squared gradients and uses them to scale each parameter update, automatically reducing steps where gradients are large and allowing relatively larger steps where they are small.