What does the depth of a neural network mean?

The depth of a neural network refers to how many layers of computation the network contains between its input and output.

In practice, people usually focus on the number of hidden or learnable layers when describing how deep a network is.

Simple example

A small network might look like:

Input → Hidden 1 → Output

A deeper network might look like:

Input → Hidden 1 → Hidden 2 → Hidden 3 → Output

The second network has greater depth because information passes through more transformations.

Why does depth matter?

Each additional layer can learn a new transformation of the representation produced by the previous layer.

This allows deep networks to build hierarchical features.

For example, in image recognition:

Pixels → Edges → Textures → Shapes → Object parts → Object

Does more depth always improve performance?

No.

Increasing depth can improve model capacity, but it can also make training harder and increase:

• Computational cost
• Memory usage
• Training time
• Risk of overfitting
• Problems such as vanishing or exploding gradients

Architectures such as residual networks use skip connections to make very deep models easier to optimize.

Depth vs width

Depth describes how many layers the network has.

Width describes how many neurons or units are present within a layer.

For the related concept, see “What does the width of a neural network mean?”.

In short, network depth describes how many successive layers transform the data before the model produces its final output.