How can you find a suitable learning rate?

A suitable learning rate is usually found by testing a range of values and choosing one that makes the loss decrease quickly without making training unstable.

There is no single learning rate that works best for every model and dataset.

Start with common values

For many optimizers, a reasonable starting point might be:

• Around 0.001 for Adam-like optimizers.
• Around 0.01 or 0.1 for SGD in some setups.

These are only starting points, not fixed rules.

Try values on a logarithmic scale

Instead of testing values that are very close together, try values such as:

0.1 → 0.01 → 0.001 → 0.0001

This makes it easier to identify the right order of magnitude.

Watch the training behavior

A useful learning rate should generally make the loss decrease at a good pace.

If the learning rate is too high, you may see:

• Oscillating loss.
• Sudden spikes.
• Divergence.

If it is too low, you may see:

• Very slow improvement.
• Tiny parameter updates.
• Training that appears stuck.

Use a learning-rate finder

A learning-rate finder gradually increases the learning rate over a short training run and records how the loss changes.

A practical choice is often near the region where the loss starts decreasing rapidly, but before it becomes unstable.

Validate the choice

Do not choose the learning rate only from training loss.

Compare validation performance as well, because the best learning rate should support both good optimization and good generalization.

Use scheduling after choosing the starting rate

Once a good initial learning rate is found, a learning-rate schedule can reduce or adjust it during training.

For the warning signs, see “What happens when the learning rate is too high?” and “What happens when the learning rate is too low?”.

In short, find a suitable learning rate by testing values across different scales, monitoring loss and validation performance, and choosing the largest rate that trains quickly without becoming unstable.