What is a gradient in deep learning?

A gradient tells us how the loss changes when a model parameter changes slightly. It is the key signal used to decide how weights and biases should be updated during training.

For a parameter w, the gradient is written as:

∂Loss / ∂w

It measures the rate of change of the loss with respect to w.

How to interpret a gradient

Suppose:

gradient = +0.8

Increasing the parameter would increase the loss, so an optimizer will usually move the parameter in the opposite direction.

If:

gradient = -0.8

increasing the parameter would reduce the loss locally, so the update direction changes accordingly.

The sign indicates direction, while the magnitude indicates how sensitive the loss is to that parameter.

Why gradients matter

A neural network may contain millions or billions of trainable parameters. Gradients provide a systematic way to determine how each parameter should change to reduce the loss.

A simplified update is:

new_parameter = old_parameter - learning_rate × gradient

The gradients themselves are typically computed using backpropagation, while an optimizer uses them to update the parameters.

For related concepts, see “What is a loss function in deep learning?”, “What are weights and biases in a neural network?”, and “How does a neural network learn?”.

In short, a gradient measures how sensitive the loss is to a model parameter and provides the direction used to improve that parameter during training.