What is the difference between weight decay and L2 regularization?

Weight decay and L2 regularization both encourage smaller weights, but they are applied differently during optimization.

With plain gradient descent, they can behave equivalently under common settings.

With adaptive optimizers such as Adam, they are generally not the same.

How does L2 regularization work?

L2 regularization adds a penalty based on squared weights to the loss:

L_total = L_data + λ × Σw²

During backpropagation, this penalty contributes an extra gradient term proportional to the weight:

gradient_total = gradient_data + λ × w

The optimizer then processes this modified gradient normally.

How does weight decay work?

Weight decay directly shrinks the parameter during the update.

A simplified form is:

w_new = w_old - learning_rate × gradient - learning_rate × weight_decay × w_old

So the shrinkage is applied separately from the loss gradient.

Why are they equivalent with standard SGD?

With ordinary SGD, adding an L2 gradient term and directly shrinking the weight can produce the same update after matching the coefficients appropriately.

That is why the terms are sometimes used interchangeably.

Why are they different with Adam?

Adam rescales gradients separately for each parameter using adaptive moment estimates.

If an L2 penalty is added to the gradient, that penalty is also transformed by Adam's adaptive scaling.

With decoupled weight decay, the shrinkage is applied separately and is not distorted by that gradient scaling.

Where does AdamW fit in?

AdamW explicitly uses decoupled weight decay.

This makes the regularization effect easier to control independently from Adam's adaptive gradient update.

For the optimizer itself, see “What is AdamW?”.

For the basic regularization concepts, see “What is L1 and L2 regularization?” and “What is weight decay in neural networks?”.

In short, L2 regularization modifies the loss and therefore the gradient, while weight decay directly shrinks the parameters; they can be equivalent with SGD but differ with adaptive optimizers such as Adam.