What is a dead neuron in a neural network?

A dead neuron in a neural network is a neuron that has effectively stopped contributing useful information because its output stays at a constant value and it receives little or no useful gradient for updating its weights.

This problem is most commonly discussed with ReLU-based networks.

How can a neuron become dead?

ReLU is defined as:

ReLU(x) = max(0, x)

If a neuron's pre-activation becomes negative for every input, ReLU outputs:

0

for all of those inputs.

Because the ReLU gradient is also zero on the negative side, that neuron may stop receiving updates that could bring it back into an active region.

Why is this a problem?

A dead neuron:

• Produces no useful activation.
• Contributes little or nothing to later layers.
• Stops learning effectively.
• Reduces the usable capacity of the network.

If many neurons die, model performance can suffer.

What can cause dead neurons?

Common causes include:

• Learning rates that are too high.
• Weight updates that push activations far into the negative region.
• Poor initialization.
• Using standard ReLU in situations where negative-side gradient flow is important.

How can dead neurons be reduced?

Possible approaches include:

• Using a smaller learning rate.
• Using good weight initialization.
• Using alternatives such as Leaky ReLU, PReLU, ELU, or other activations with nonzero negative-side gradients.

The closely related issue is explained in “What is the dying ReLU problem?”.

In short, a dead neuron is a neuron that becomes effectively inactive and stops learning, often because ReLU keeps its output and gradient at zero for all inputs.