What are Xavier and He initialization?

Xavier initialization and He initialization are methods for choosing the starting scale of neural-network weights so activations and gradients remain reasonably stable across layers.

Xavier initialization

Xavier, also called Glorot initialization, is commonly used with activation functions such as tanh or sigmoid-like activations.

Its weight scale depends on both the number of inputs and outputs of a layer:

variance ≈ 2 / (fan_in + fan_out)

Here, fan_in is the number of incoming connections and fan_out is the number of outgoing connections.

He initialization

He initialization is designed especially for ReLU and ReLU-like activations.

A common form uses:

variance ≈ 2 / fan_in

Because ReLU sets many negative activations to zero, He initialization uses a larger variance than Xavier to help preserve signal strength.

Main difference

• Xavier: commonly suited to tanh or sigmoid-style activations.
• He: commonly suited to ReLU and related activations.

Both methods try to reduce problems such as vanishing or exploding activations and gradients at the start of training.

For the broader concept, see “What is weight initialization in neural networks?” and “Why is weight initialization important in deep learning?”.

In short, Xavier and He initialization choose weight scales based on layer size, with Xavier commonly used for tanh-like activations and He commonly used for ReLU-like activations.