How does the Adam optimizer work?

The Adam optimizer works by combining momentum with adaptive learning rates for each parameter.

It tracks both the average gradient direction and the average squared gradient magnitude, then uses them to scale each update.

Step 1: Calculate the gradient

After backpropagation, Adam receives the current gradient:

g_t

This gradient shows how the loss changes with respect to each parameter.

Step 2: Update the first moment

Adam keeps an exponential moving average of recent gradients:

m_t = β₁ × m_(t-1) + (1 - β₁) × g_t

This behaves similarly to momentum and helps smooth the update direction.

Step 3: Update the second moment

Adam also tracks an exponential moving average of squared gradients:

v_t = β₂ × v_(t-1) + (1 - β₂) × g_t²

This measures the recent scale of the gradients.

Step 4: Correct the initial bias

Because m_t and v_t start at zero, their early values are biased toward zero.

Adam corrects this using:

m̂_t = m_t / (1 - β₁^t)

v̂_t = v_t / (1 - β₂^t)

Step 5: Update the parameter

The parameter is then updated using:

w_new = w_old - learning_rate × m̂_t / (sqrt(v̂_t) + ε)

The small value ε prevents division by zero.

What does this achieve?

Adam effectively gives each parameter its own adaptive step size.

Parameters with consistently large gradients receive relatively smaller updates, while parameters with smaller gradients can receive relatively larger updates.

At the same time, the first-moment estimate provides momentum-like smoothing.

Typical defaults are:

β₁ = 0.9

β₂ = 0.999

ε ≈ 1e-8

For the overview, see “What is the Adam optimizer?”.

In short, Adam combines a moving average of gradients with a moving average of squared gradients, applies bias correction, and uses both to produce adaptive, momentum-like parameter updates.