Why are neural network weights initialized randomly?

Neural network weights are initialized randomly to break symmetry between neurons so they can learn different features during training.

If all neurons in the same layer started with identical weights, they would produce identical outputs, receive identical gradients, and remain identical after every update.

What does symmetry mean here?

Suppose two neurons in the same layer begin with exactly the same weights and bias.

For the same input, they compute the same value:

Neuron 1 output = Neuron 2 output

During backpropagation, they also receive the same gradient.

So after an update:

Neuron 1 weights = Neuron 2 weights

They continue behaving like copies of each other instead of learning different patterns.

How does random initialization help?

Small random differences give neurons different starting points.

This allows them to:

• Produce different activations.
• Receive different gradients.
• Learn different features.
• Use the layer's full representational capacity.

Does random mean completely arbitrary?

No.

Good initialization methods control the scale of the random values so activations and gradients do not become too large or too small.

Common methods include:

• Xavier or Glorot initialization.
• He initialization.

These methods choose the random scale based on the number of connections entering or leaving a layer.

Why not initialize every weight to zero?

Zero initialization does not break symmetry between neurons in the same layer.

This specific problem is covered in “What happens if all neural network weights are initialized to zero?”.

For initialization methods themselves, see “What are Xavier and He initialization?”.

In short, weights are initialized randomly so different neurons start differently and can learn different features, while carefully chosen initialization scales help keep training stable.