What is the symmetry problem in neural networks?

The symmetry problem in neural networks occurs when multiple neurons start with identical parameters and therefore learn the same features instead of different ones.

This commonly happens if all weights in a layer are initialized to the same value.

What does symmetry mean?

Suppose two neurons in the same hidden layer have identical weights and biases.

For the same input, they produce identical outputs:

Neuron 1 output = Neuron 2 output

What happens during backpropagation?

Because the neurons behave identically, they also receive identical gradients.

After every update:

Neuron 1 weights = Neuron 2 weights

So the neurons remain copies of each other throughout training.

Why is this a problem?

A layer may contain many neurons, but if they all learn the same thing, the network does not benefit from having multiple independent neurons.

This reduces the effective capacity of the model and prevents different neurons from specializing in different patterns.

How is the symmetry problem avoided?

The standard solution is to initialize weights with different random values.

Methods such as:

• Xavier or Glorot initialization
• He initialization

both break symmetry while also controlling the scale of the initial weights.

What about biases?

Biases can often be initialized to zero because symmetry is already broken when the weights connected to different neurons are initialized differently.

For the practical consequence of zero initialization, see “What happens if all neural network weights are initialized to zero?”.

In short, the symmetry problem happens when neurons start identically, receive identical gradients, and continue learning the same features, which is why neural network weights are normally initialized differently.