What happens if all neural network weights are initialized to zero?

If all neural network weights are initialized to zero, neurons in the same layer start identically and continue learning the same thing.

This creates a symmetry problem that prevents the layer from using different neurons to learn different features.

What happens during the forward pass?

If two neurons have the same inputs, zero weights, and the same bias, they produce the same output.

For example:

Neuron 1 output = Neuron 2 output

What happens during backpropagation?

Because the neurons behave identically, they also receive identical gradients.

After an update:

Neuron 1 weights = Neuron 2 weights

So the neurons remain identical after every training step.

Why is this a problem?

A layer with many identical neurons behaves as if it had far fewer independent neurons.

The network loses much of its ability to learn diverse features.

Does zero initialization always prevent learning?

Not every parameter must be random.

Biases are often safely initialized to zero because different neurons can still behave differently when their weights are initialized differently.

The main problem is initializing all weights in a layer to the same value, especially zero.

What should be used instead?

Weights are usually initialized with small random values using methods such as:

• Xavier or Glorot initialization.
• He initialization.

These methods both break symmetry and help keep activations and gradients at useful scales.

For the reason random initialization is used, see “Why are neural network weights initialized randomly?”. The underlying issue is covered next in “What is the symmetry problem in neural networks?”.

In short, initializing all weights to zero causes neurons in the same layer to remain identical, preventing them from learning different features and wasting much of the network's capacity.