What is gradient descent?

Gradient descent is an optimization method used to reduce a model's loss by repeatedly adjusting its trainable parameters in the direction that lowers the loss.

A simplified update is:

new_parameter = old_parameter - learning_rate × gradient

The gradient tells us the direction in which the loss increases, so gradient descent moves in the opposite direction.

Simple intuition

Imagine the loss as a landscape and the current model parameters as a point on that landscape.

Higher loss
    /\
   /  \      ● current position
  /    \    ↙
 /      \__
          \___ Lower loss

Gradient descent uses the local slope to decide which way to move toward a lower-loss region.

How it fits into training

Forward pass
   ↓
Loss
   ↓
Backpropagation computes gradients
   ↓
Gradient descent updates parameters

For the gradient itself, see “What is a gradient in deep learning?”. For how gradients are computed, see “How does backpropagation work?”.

Role of the learning rate

The learning rate controls the size of each update.

• Too small: training can be very slow.
• Too large: updates may overshoot useful regions or become unstable.

A dedicated answer, “What is a learning rate?”, covers this in more detail.

In short, gradient descent improves a neural network by repeatedly moving its parameters in the direction that reduces the loss.