What is learning rate decay?

Learning rate decay is the practice of gradually reducing the learning rate during neural network training.

The idea is to use larger updates early in training and smaller, more precise updates later.

Why decay the learning rate?

At the beginning of training, the model is usually far from a good solution.

A relatively larger learning rate can help it make faster progress.

Later in training, smaller updates can help the optimizer:

• Fine-tune the model.
• Reduce oscillation around a minimum.
• Improve convergence.

Simple idea

Training might begin with:

learning rate = 0.01

and gradually reduce it to:

0.001

and later:

0.0001

The exact pattern depends on the chosen schedule.

Common learning rate decay methods

Common approaches include:

• Step decay: reduce the learning rate after fixed intervals.
• Exponential decay: multiply it by a constant factor over time.
• Cosine annealing: reduce it according to a cosine-shaped curve.
• Performance-based decay: reduce it when a validation metric stops improving.

Is learning rate decay the same as learning rate scheduling?

Learning rate decay is a type of learning rate scheduling.

A learning rate schedule is the broader strategy for changing the learning rate during training, while decay specifically refers to reducing it over time.

Why can it improve training?

A constant learning rate may be useful early but too large near a good solution.

Decay allows the optimizer to move quickly at first and then take smaller steps as training progresses.

For the broader concept, see “What is learning rate scheduling?”. The next articles cover specific decay strategies such as “What is step learning rate decay?” and “What is exponential learning rate decay?”.

In short, learning rate decay gradually reduces the learning rate so training can make fast progress early and more precise updates later.