What is ReLU and why is it widely used?

ReLU (Rectified Linear Unit) is an activation function commonly used in neural networks, especially in hidden layers. It keeps positive inputs unchanged and converts negative inputs to zero.

Its definition is:

ReLU(x) = max(0, x)

So:

ReLU(-4) = 0
ReLU(-1) = 0
ReLU(0)  = 0
ReLU(2)  = 2
ReLU(7)  = 7

How ReLU behaves

ReLU is piecewise linear:

          y
          ↑
          |
          |        /
          |       /
          |      /
----------+-----/----------→ x
          |

For negative inputs, the output is zero. For positive inputs, the output increases linearly with the input.

Mathematically:

f(x) = 0    if x ≤ 0
f(x) = x    if x > 0

Why is ReLU widely used?

ReLU became popular because it combines a useful nonlinearity with a very simple computation.

1. It is computationally inexpensive

ReLU only needs a comparison with zero:

PYTHON
def relu(x):
    return max(0.0, x)

There is no exponential or other expensive nonlinear operation in the basic function.

2. It helps deep networks model nonlinear relationships

Although ReLU is linear on each side of zero, it is nonlinear as a whole. Placing ReLU between learned linear transformations prevents a deep network from collapsing into a single linear transformation.

A simplified hidden layer can be written as:

h = ReLU(Wx + b)

The combination of learned weights and ReLU allows the network to build piecewise-linear decision functions and increasingly complex representations across layers.

3. Its gradient is simple for positive inputs

For positive inputs:

ReLU'(x) = 1

For negative inputs:

ReLU'(x) = 0

At exactly zero, the derivative is not mathematically unique, so deep-learning frameworks use a practical convention.

Because the derivative is 1 in the positive region, gradients can pass through active ReLU units without being repeatedly shrunk by the activation itself.

4. It produces sparse activations

Negative pre-activation values become zero. Therefore, for a given input, some neurons may output exactly zero.

For example:

Before ReLU: [-2.4, 1.2, -0.7, 3.5]
After ReLU:  [ 0.0, 1.2,  0.0, 3.5]

This creates sparse intermediate activations, where only a subset of units are active for that input.

Example inside a neural network

Suppose a neuron computes:

z = 2.3

After ReLU:

ReLU(2.3) = 2.3

If another neuron computes:

z = -1.8

then:

ReLU(-1.8) = 0

A layer may therefore transform values like this:

[-1.8, 2.3, -0.5, 4.1]
          ↓ ReLU
[ 0.0, 2.3,  0.0, 4.1]

What is the dying ReLU problem?

A ReLU neuron can become effectively inactive if its input remains negative for all or most training examples.

In that region:

output = 0
gradient = 0

If the gradient remains zero, the parameters feeding that unit may stop receiving useful updates through that activation path.

This is commonly called the dying ReLU problem.

It can be reduced by appropriate initialization, learning-rate choices, normalization, or by using variants such as Leaky ReLU when suitable.

ReLU vs Leaky ReLU

Standard ReLU:

x < 0 → 0

Leaky ReLU:

x < 0 → αx

where α is a small positive constant.

Leaky ReLU preserves a small gradient in the negative region, which can help prevent units from becoming permanently inactive.

PyTorch example

PYTHON
import torch
import torch.nn as nn

x = torch.tensor([-2.0, -0.5, 0.0, 1.5, 3.0])
relu = nn.ReLU()

print(relu(x))

Output:

tensor([0.0000, 0.0000, 0.0000, 1.5000, 3.0000])

In short, ReLU is widely used because it is simple, fast, introduces nonlinearity, and provides favorable gradient behavior for positive activations. Its main limitation is that units can become inactive when they remain in the negative region.