Why does increasing neural network depth sometimes reduce performance?

Increasing a neural network's depth can reduce performance because extra layers make optimization harder and may increase overfitting, even though they give the model more capacity. More layers do not automatically produce a better model.

Why can performance get worse?

• Optimization difficulty: Very deep networks can be harder to train. In plain networks, adding layers may even increase *training error*; this is known as the degradation problem.
• Unstable gradients: Gradients can shrink or grow as they pass through many layers, making earlier layers difficult to update reliably.
• Overfitting: A larger model may fit training examples well but generalize poorly, especially with limited data.
• Greater training demands: Deeper models often need more computation, careful initialization, and suitable learning-rate schedules. With the same training budget, they may remain undertrained.

How do you identify the cause?

• Training error increases after adding layers: Suspect optimization or degradation, not just overfitting.
• Training error improves but validation error worsens: Suspect overfitting or a generalization problem.
• Both errors remain high: Check optimization settings, training duration, and model suitability.

What can help?

• Use residual or skip connections to provide easier information and gradient paths.
• Apply appropriate normalization and weight initialization.
• Tune the learning rate and give the model enough training time.
• Use regularization or more data when validation performance indicates overfitting.
• Choose depth based on validation results rather than assuming deeper is always better.

Example: A 50-layer plain network may train worse than a 20-layer network, even on the training set. Residual connections make it easier for added layers to learn small corrections—or approximate an identity mapping—rather than disrupting useful features.

For related explanations, see “How does network depth affect neural network performance?” and “Why are residual networks easier to train?”.

In short: More depth increases representational capacity, but performance can fall when optimization becomes harder, training is insufficient, or the model overfits.

Research: Deep Residual Learning for Image Recognition.