What is the GELU activation function?

GELU (Gaussian Error Linear Unit) is a smooth nonlinear activation function that scales an input based on how large it is rather than simply turning negative values off.

It is widely used in modern deep learning architectures, especially Transformer-based models.

Formula

GELU is commonly written as:

GELU(x) = x · Φ(x)

where Φ(x) is the cumulative distribution function of the standard normal distribution.

A common approximation is:

GELU(x) ≈ 0.5x[1 + tanh(√(2/π)(x + 0.044715x³))]

How does GELU behave?

• Large positive inputs pass through almost unchanged.
• Large negative inputs are pushed close to zero.
• Small negative inputs are reduced smoothly rather than being cut off abruptly.

How is GELU different from ReLU?

ReLU uses a hard threshold:

ReLU(x) = max(0, x)

So all negative values become zero.

GELU uses a smooth weighting instead, allowing some small negative values to remain.

Why is GELU useful?

Its smooth behavior can provide stable gradient flow and works well in large deep networks.

This is one reason GELU is commonly used in Transformer architectures.

Main limitation

GELU is computationally more expensive than ReLU because its exact form involves the Gaussian cumulative distribution function, although efficient approximations are widely used.

For comparison, see “What is ReLU and why is it widely used?” and “What is the Swish activation function?”.

In short, GELU is a smooth activation function that softly scales inputs instead of applying a hard cutoff, making it especially popular in modern deep neural networks such as Transformers.