How does a residual block work?

A residual block works by processing an input through several layers while also carrying the original input along a shortcut path. The two paths are combined to produce the block's output.

Step-by-step flow

1. Receive the input: Let the block's input be x.
2. Transform it: The main path applies layers such as convolutions, normalization, and activations to compute F(x).
3. Keep a shortcut: The skip path passes x forward, usually without changing it.
4. Combine both paths: Add the shortcut and learned transformation:

y = F(x) + x

1. Produce the output: Some designs apply an activation after the addition; others, such as pre-activation ResNets, use a different ordering.

Example

Suppose the main path learns a small correction F(x) to image features. Adding x preserves the earlier features while incorporating that correction. If the input and output dimensions differ, the shortcut can use a projection, often a 1×1 convolution, before addition.

How does this help backpropagation?

The shortcut gives gradients a direct route around the main path. With an identity shortcut, the derivative of x + F(x) includes an identity term, helping gradients reach earlier layers even when the main path is difficult to optimize.

For the block's definition, see “What is a residual block?”. For its training benefit, see “How do skip connections help train deep neural networks?”.

In short: A residual block learns a change to its input and adds that change back to the original signal, helping information and gradients travel through deep networks.