How are gradients calculated in a neural network?

Gradients in a neural network are calculated by measuring how much the loss changes when each trainable parameter changes.

This is done during backpropagation using derivatives and the chain rule.

Step 1: Perform a forward pass

The input moves through the network to produce a prediction:

Input → Layers → Prediction

The prediction is then compared with the target using a loss function.

Step 2: Start from the loss

Backpropagation begins by calculating how the loss changes with respect to the network's output.

This gives the first gradient signal.

Step 3: Move backward through the network

For each layer, the chain rule combines:

• The gradient arriving from the next layer.
• The derivative of the activation function.
• The derivative of the layer's weighted computation.

A simplified form is:

gradient = incoming gradient × local derivative

Example for one weight

Suppose:

z = wx + b

a = f(z)

L = loss(a)

Then the gradient of the loss with respect to w is:

dL/dw = (dL/da) × (da/dz) × (dz/dw)

Since:

dz/dw = x

the chain rule connects the loss all the way back to the weight.

What happens across many layers?

The same process is repeated layer by layer from the output toward the input.

Modern deep learning frameworks usually calculate these derivatives automatically using automatic differentiation.

What happens after gradients are calculated?

An optimizer uses the gradients to update the parameters.

For example:

w_new = w_old - learning_rate × gradient

For the underlying calculus rule, see “What is the chain rule in backpropagation?”.

In short, gradients are calculated by propagating the loss backward through the network and repeatedly applying the chain rule to determine how each parameter affects the loss.