What is Parametric ReLU (PReLU)?

Parametric ReLU (PReLU) is a variation of ReLU in which the slope for negative inputs is learned during training instead of being fixed manually.

Formula

PReLU is commonly written as:

f(x) = x, if x > 0

f(x) = ax, if x ≤ 0

where a is a learnable parameter.

How is PReLU different from Leaky ReLU?

Both allow a small nonzero output for negative inputs.

The difference is:

• Leaky ReLU: uses a fixed negative slope such as 0.01.
• PReLU: learns the negative slope from the training data.

Why is PReLU useful?

Like Leaky ReLU, PReLU helps reduce the dying ReLU problem by allowing gradients to flow when the input is negative.

Because the negative slope is learned, the network can adapt this behavior to the task instead of relying on a manually chosen value.

What is learned during training?

The parameter a is updated through backpropagation along with the network's weights and biases.

Depending on the implementation, a layer may use one shared parameter or separate parameters for different channels or units.

Main trade-off

PReLU introduces additional learnable parameters, although usually only a small number.

It can provide more flexibility than Leaky ReLU, but that flexibility does not guarantee better performance for every problem.

For the fixed-slope version, see “What is Leaky ReLU?”.

In short, PReLU is a ReLU variant that learns the slope used for negative inputs, allowing the activation function itself to adapt during training.