What is Leaky ReLU?

Leaky ReLU is a variation of the ReLU activation function that allows a small negative output instead of returning zero for all negative inputs.

Formula

A common form is:

f(x) = x, if x > 0

f(x) = αx, if x ≤ 0

where α is a small positive constant such as 0.01.

How is it different from ReLU?

Standard ReLU uses:

f(x) = max(0, x)

So every negative input becomes zero.

Leaky ReLU keeps a small slope for negative inputs:

Negative input → Small negative output

Why is Leaky ReLU useful?

It helps reduce the dying ReLU problem, where some neurons can get stuck producing zero and stop receiving useful gradients.

Because Leaky ReLU keeps a small gradient on the negative side, neurons have a better chance of continuing to learn.

Main advantage

• Simple and computationally efficient.
• Preserves ReLU's strong positive-side behavior.
• Allows some gradient flow for negative inputs.

Main limitation

The negative slope α is usually chosen manually and may not be optimal for every task.

A related activation called Parametric ReLU (PReLU) learns this negative slope from data instead of fixing it.

For the original activation, see “What is ReLU and why is it widely used?”.

In short, Leaky ReLU is ReLU with a small nonzero slope for negative inputs, helping prevent neurons from becoming permanently inactive.