What is the sigmoid activation function?

The sigmoid activation function maps any real-valued input to a value between 0 and 1. It is especially useful when a model output needs to represent a probability-like score for a binary outcome.

Its formula is:

sigmoid(x) = 1 / (1 + e^(-x))

The function has an S-shaped curve:

Output
1.0 |                 ______
    |              __/
0.5 |------------/
    |         __/
0.0 |_______/
    +------------------------→ Input

How sigmoid transforms values

Some example values are:

sigmoid(-5) ≈ 0.0067
sigmoid(-1) ≈ 0.2689
sigmoid(0)  = 0.5
sigmoid(1)  ≈ 0.7311
sigmoid(5)  ≈ 0.9933

Large negative inputs produce values close to 0, while large positive inputs produce values close to 1.

A value of 0 always maps to:

sigmoid(0) = 0.5

Why is sigmoid useful for binary classification?

For a binary classifier, a neural network can produce a single raw score called a logit.

For example:

Neural network output logit = 2.2
            ↓
         Sigmoid
            ↓
         0.9002

The transformed value can be interpreted as a probability-like prediction for the positive class when the model and loss are configured appropriately.

For example:

0.90 → high confidence for the positive class
0.12 → low confidence for the positive class

A classification threshold such as 0.5 may then be applied when a hard class decision is required.

Sigmoid derivative

The derivative of sigmoid has a convenient form:

σ'(x) = σ(x)(1 - σ(x))

where σ(x) is the sigmoid output.

For example, if:

σ(x) = 0.5

then:

σ'(x) = 0.5 × (1 - 0.5)
       = 0.25

The derivative reaches its maximum value at the center of the curve and becomes much smaller toward either extreme.

What is saturation?

When the input is very large or very negative, sigmoid approaches 1 or 0.

For example:

x = 10   → sigmoid(x) ≈ 0.99995
x = -10  → sigmoid(x) ≈ 0.00005

In these regions, the derivative becomes very small.

This behavior is called saturation.

During backpropagation, very small derivatives can reduce gradient magnitude as gradients move through many layers. This is one reason sigmoid is generally not the default choice for hidden layers in modern deep neural networks.

Sigmoid in the output layer

Sigmoid remains very useful in output layers where each output represents an independent binary decision.

Examples include:

• Spam vs. not spam
• Fraud vs. legitimate transaction
• Disease present vs. absent
• Multiple independent labels in multilabel classification

For multilabel classification, several output neurons can each use sigmoid independently:

Input image
    ↓
Neural network
    ↓
[0.92, 0.15, 0.81]
  cat   dog   outdoor

Unlike softmax, these values do not need to sum to 1 because multiple labels can be true at the same time.

Sigmoid and binary cross-entropy

Sigmoid is commonly paired conceptually with binary cross-entropy loss.

However, in practical deep-learning frameworks, it is often better to use a loss function that accepts raw logits and performs the sigmoid transformation internally for improved numerical stability.

For example, in PyTorch:

PYTHON
import torch
import torch.nn as nn

logits = torch.tensor([2.0, -1.0, 0.5])
targets = torch.tensor([1.0, 0.0, 1.0])

loss_fn = nn.BCEWithLogitsLoss()
loss = loss_fn(logits, targets)

print(loss)

BCEWithLogitsLoss combines the sigmoid operation with binary cross-entropy in a numerically stable implementation.

If probabilities are needed for inference, sigmoid can be applied explicitly:

PYTHON
probabilities = torch.sigmoid(logits)
print(probabilities)

Main strengths and limitations

• Output range: 0 to 1.
• Shape: Smooth S-shaped curve.
• Common use: Binary or multilabel probability-like outputs.
• Main strength: Naturally maps unrestricted scores into a probability-like range.
• Main limitation: Saturation causes very small gradients at extreme input values.

In short, sigmoid converts an unrestricted real-valued score into a value between 0 and 1, making it especially useful for binary and multilabel outputs. Its main drawback is saturation, which makes it less suitable for many hidden-layer applications.