What is exponential learning rate decay?

Exponential learning rate decay is a scheduling method that reduces the learning rate continuously by multiplying it by a decay factor as training progresses.

Unlike step decay, which drops the learning rate at fixed intervals, exponential decay changes it more smoothly.

How does it work?

A common form is:

learning_rate = initial_learning_rate × decay_rate^t

where t represents training progress, such as epochs or steps.

Simple example

Suppose:

initial learning rate = 0.01

and the learning rate is multiplied by:

0.9

after each interval.

Then it becomes approximately:

0.0100 → 0.0090 → 0.0081 → 0.00729 → ...

The learning rate keeps shrinking by the same proportion.

Why use exponential decay?

It provides:

• Larger updates early in training.
• Gradually smaller updates later.
• A smooth and predictable reduction in learning rate.

This can help the optimizer move quickly at first and make more precise adjustments as training progresses.

What controls the decay?

The main settings are:

• Initial learning rate: the starting value.
• Decay rate: how quickly the learning rate shrinks.
• Decay interval: how often the decay is applied.

If the decay is too aggressive, the learning rate may become too small too early.

If it is too slow, the model may continue making overly large updates late in training.

How is it different from step decay?

Step decay reduces the learning rate in sudden jumps.

Exponential decay reduces it progressively according to an exponential curve.

For the broader concept, see “What is learning rate decay?”.

In short, exponential learning rate decay gradually multiplies the learning rate by a fixed decay factor, producing progressively smaller optimization steps throughout training.