What is an optimizer in deep learning?

An optimizer is the algorithm that updates a neural network's trainable parameters using the gradients computed during training.

Backpropagation calculates the gradients. The optimizer uses those gradients to decide how the weights and biases should change.

Loss
  ↓
Backpropagation
  ↓
Gradients
  ↓
Optimizer
  ↓
Updated parameters

Basic idea

A simple parameter update looks like:

new_parameter = old_parameter - learning_rate × gradient

The optimizer controls how this update is applied. More advanced optimizers can also use information from previous updates or adapt the step size for individual parameters.

Common optimizers

• SGD (Stochastic Gradient Descent): updates parameters directly from gradients, often with optional momentum.
• Adam: keeps running estimates of gradient behavior and adapts the update size for each parameter.

The dedicated answer “What is the difference between SGD and Adam?” covers that comparison in detail.

Why does the optimizer matter?

The optimizer affects how quickly and how reliably the model moves toward a lower-loss solution. A poor optimizer configuration can make training slow or unstable, even when the network architecture is otherwise suitable.

The optimizer also depends heavily on the learning rate. See “What is a learning rate?” for that concept.

For the optimization principle behind these updates, see “What is gradient descent?”.

In short, an optimizer is the mechanism that turns gradients into parameter updates during neural-network training.