Why is learning rate warm-up used in deep neural networks?

Learning rate warm-up is used in deep neural networks to prevent unstable parameter updates during the early stages of training.

At the beginning of training, the model's weights, gradients, and optimizer statistics may not yet be well calibrated.

Starting immediately with a large learning rate can therefore make the first few updates too aggressive.

What does warm-up change?

Instead of using the full learning rate from the first step, training starts with a much smaller value and gradually increases it.

For example:

0.00001 → 0.0001 → 0.0005 → 0.001

Once the target learning rate is reached, the normal learning-rate schedule continues.

Why does this help?

Warm-up gives the model time to enter a more stable training regime before large updates are allowed.

It can help reduce:

• Sudden spikes in loss.
• Unstable gradients.
• Overshooting early in training.
• Divergence caused by aggressive initial updates.

Why is it useful for very deep networks?

In deep networks, small changes in early layers can propagate through many later layers.

Because of this, unstable updates near the start of training can have a large effect on the entire model.

Warm-up reduces the size of those early updates.

Why is it common with large batch sizes?

Large-batch training often uses larger peak learning rates.

Jumping directly to that high value can be unstable, so warm-up gradually moves the optimizer toward the intended learning rate.

Why is it common in Transformer training?

Transformer models are often large and sensitive to optimization settings, especially early in training.

Warm-up is therefore frequently combined with schedules such as linear decay or cosine decay.

For the basic mechanism, see “What is a warm-up learning rate?”.

In short, learning rate warm-up improves early training stability by starting with small updates and gradually increasing the learning rate before normal training begins.