What are the most common activation functions?

Activation functions transform the values produced by neurons or layers in a neural network. The most common activation functions include ReLU, Leaky ReLU, sigmoid, tanh, softmax, GELU, SiLU/Swish, and ELU. Different functions are useful in different architectures and parts of a network.

1. ReLU

Rectified Linear Unit (ReLU) is one of the most widely used activation functions for hidden layers.

ReLU(x) = max(0, x)

Examples:

x = -3 → 0
x =  0 → 0
x =  4 → 4

Advantages include computational simplicity and good optimization behavior in many deep networks. A limitation is that a neuron can become inactive if it remains in the negative-input region, where the gradient is zero.

2. Leaky ReLU

Leaky ReLU modifies ReLU by allowing a small nonzero slope for negative inputs.

f(x) = x        if x > 0
f(x) = αx       if x ≤ 0

Here, α is a small positive value.

This helps retain gradient information for negative inputs and can reduce the inactive-neuron problem associated with standard ReLU.

3. Sigmoid

The sigmoid function maps a real number to a value between 0 and 1.

sigmoid(x) = 1 / (1 + e⁻ˣ)

Its shape is approximately:

0 ─────╮
       │╲
       │ ╲───── 1

Sigmoid is commonly used for binary classification outputs or independent probability-like outputs.

A major limitation is saturation: for large positive or negative inputs, its derivative becomes very small, which can contribute to vanishing gradients in deep networks.

4. Tanh

The hyperbolic tangent (tanh) maps inputs to values between approximately -1 and 1.

tanh(x)

Unlike sigmoid, tanh is centered around zero. It has historically been common in recurrent neural networks, although it can also suffer from saturation and vanishing gradients.

5. Softmax

Softmax transforms a vector of logits into a probability distribution.

For class i:

softmax(zᵢ) = exp(zᵢ) / Σ exp(zⱼ)

For example:

Logits
[2.1, 0.8, -0.4]
      ↓
Softmax
      ↓
[0.74, 0.20, 0.06]

The output values are non-negative and sum to 1. Softmax is commonly used when interpreting multiclass model outputs as probabilities across mutually exclusive classes.

6. GELU

Gaussian Error Linear Unit (GELU) is a smooth activation widely used in Transformer-based architectures.

A common definition is:

GELU(x) = xΦ(x)

where Φ(x) is the cumulative distribution function of the standard normal distribution.

Unlike ReLU, which sharply sets negative inputs to zero, GELU smoothly scales inputs according to their value.

GELU is used in many modern Transformer architectures.

7. SiLU / Swish

SiLU (Sigmoid Linear Unit), often called Swish, is defined as:

SiLU(x) = x × sigmoid(x)

It is smooth and allows some negative outputs. SiLU is used in a number of modern deep-learning architectures, including some computer-vision and language models.

8. ELU

Exponential Linear Unit (ELU) behaves linearly for positive values and uses an exponential curve for negative values.

f(x) = x                 if x > 0
f(x) = α(exp(x) - 1)     if x ≤ 0

Unlike standard ReLU, ELU can produce negative outputs and has a nonzero gradient over much of its negative region.

Quick comparison

• ReLU: Outputs values from 0 upward; simple and efficient for many hidden layers.
• Leaky ReLU: Allows a small negative slope, helping preserve gradients for negative inputs.
• Sigmoid: Outputs values between 0 and 1; commonly used for binary probability-like outputs.
• Tanh: Outputs values between -1 and 1; zero-centered and historically common in recurrent networks.
• Softmax: Produces class probabilities that sum to 1; common for multiclass outputs.
• GELU: A smooth activation widely used in Transformer architectures.
• SiLU / Swish: Smooth and non-monotonic; used in many modern deep networks.
• ELU: Uses a smooth negative region instead of setting all negative values to zero.

Which activation function should you use?

There is no activation function that is best for every network.

Common starting points are:

General hidden layers     → ReLU or a related variant
Transformer architectures → GELU or SiLU, depending on architecture
Binary output             → Sigmoid
Multiclass probabilities  → Softmax
Regression output         → Often linear/no output activation

The final choice depends on the architecture, training objective, output requirements, and empirical performance.

PyTorch example

PYTHON
import torch.nn as nn

relu = nn.ReLU()
leaky_relu = nn.LeakyReLU(negative_slope=0.01)
sigmoid = nn.Sigmoid()
tanh = nn.Tanh()
softmax = nn.Softmax(dim=1)
gelu = nn.GELU()
silu = nn.SiLU()
elu = nn.ELU()

In short, ReLU and its variants are common in general neural networks, sigmoid and softmax are frequently used for specific output layers, and GELU or SiLU are common choices in many modern deep-learning architectures.