What is the Swish activation function?

The Swish activation function is a smooth nonlinear activation function defined by multiplying an input by its sigmoid value.

Formula

Swish is commonly written as:

f(x) = x · sigmoid(x)

where:

sigmoid(x) = 1 / (1 + e^-x)

How does Swish behave?

• Large positive inputs behave approximately like x.
• Large negative inputs approach 0.
• Small negative inputs can remain slightly negative instead of being forced to zero.

This makes Swish smooth and non-monotonic, meaning its output does not always increase as the input increases.

How is Swish different from ReLU?

ReLU is:

f(x) = max(0, x)

ReLU completely removes negative values.

Swish instead allows some negative outputs and changes smoothly across zero.

Why can Swish be useful?

Because it is smooth and preserves some negative information, Swish can provide better gradient flow than ReLU in some deep networks.

It has been used successfully in architectures such as EfficientNet.

Main limitation

Swish is more computationally expensive than ReLU because it requires evaluating a sigmoid function.

It also does not guarantee better performance for every architecture or dataset.

A closely related activation is SiLU (Sigmoid Linear Unit). In most modern deep learning libraries, SiLU and Swish with a fixed coefficient of 1 refer to the same function.

For comparison with other activations, see “What is ReLU and why is it widely used?”.

In short, Swish multiplies the input by its sigmoid value, creating a smooth activation that keeps some negative values and can improve optimization in some deep networks.