What is the difference between shallow and deep neural networks?

The main difference between shallow and deep neural networks is the number of hidden layers used to transform the input before producing an output.

A shallow network has only a small number of hidden layers, while a deep neural network has many.

Shallow neural network

A shallow neural network typically has:

Input layer → One or a few hidden layers → Output layer

It can learn useful nonlinear relationships, but its ability to build complex hierarchical representations is more limited.

Deep neural network

A deep neural network contains multiple hidden layers:

Input → Hidden 1 → Hidden 2 → Hidden 3 → ... → Output

Each layer can transform the representation learned by the previous layer.

This allows deeper networks to learn features in stages.

For example, in image recognition:

Pixels → Edges → Shapes → Object parts → Object

Why can deeper networks be more powerful?

Depth allows a network to represent some complex functions more efficiently by composing many simpler transformations.

Deep networks are especially useful for complex data such as:

• Images
• Audio
• Text
• Video

Is a deeper network always better?

No.

Adding more layers can also make training harder and may increase:

• Computational cost
• Memory usage
• Training time
• Risk of overfitting
• Optimization problems such as vanishing or exploding gradients

Techniques such as normalization, better initialization, and residual connections help make very deep networks easier to train.

There is no universal number of layers that officially separates a shallow network from a deep one; the terms are relative.

For the basic concept of network depth, see “What does the depth of a neural network mean?”.

In short, shallow networks use relatively few hidden layers, while deep networks use many layers to learn increasingly complex hierarchical representations.