In the context of artificial neural networks, the rectifier or ReLU (rectified linear unit) activation function is an activation function defined as the positive part of its argument:

where x is the input to a neuron. This is also known as a ramp function and is analogous to half-wave rectification in electrical engineering. This activation function was introduced by Kunihiko Fukushima in 1969 in the context of visual feature extraction in hierarchical neural networks. It was later argued that it  has strong biological motivations and mathematical justifications. In 2011 it was found to enable better training of deeper networks, compared to the widely used activation functions prior to 2011, e.g., the logistic sigmoid (which is inspired by probability theory; see logistic regression) and its more practical counterpart, the hyperbolic tangent. The rectifier is, , the most popular activation function for deep neural networks.

Rectified linear units find applications in computer vision and speech recognition using deep neural nets and computational neuroscience.

Advantages 

 Sparse activation: For example, in a randomly initialized network, only about 50% of hidden units are activated (have a non-zero output).
 Better gradient propagation: Fewer vanishing gradient problems compared to sigmoidal activation functions that saturate in both directions.
 Efficient computation: Only comparison, addition and multiplication.
 Scale-invariant: .

Rectifying activation functions were used to separate specific excitation and unspecific inhibition in the neural abstraction pyramid, which was trained in a supervised way to learn several computer vision tasks. In 2011, the use of the rectifier as a non-linearity has been shown to enable training deep supervised neural networks without requiring unsupervised pre-training. Rectified linear units, compared to sigmoid function or similar activation functions, allow faster and effective training of deep neural architectures on large and complex datasets.

Potential problems 
 Non-differentiable at zero; however, it is differentiable anywhere else, and the value of the derivative at zero can be arbitrarily chosen to be 0 or 1.
 Not zero-centered.
 Unbounded.
 Dying ReLU problem: ReLU (rectified linear unit) neurons can sometimes be pushed into states in which they become inactive for essentially all inputs. In this state, no gradients flow backward through the neuron, and so the neuron becomes stuck in a perpetually inactive state and "dies". This is a form of the vanishing gradient problem. In some cases, large numbers of neurons in a network can become stuck in dead states, effectively decreasing the model capacity. This problem typically arises when the learning rate is set too high. It may be mitigated by using leaky ReLUs instead, which assign a small positive slope for x < 0; however, the performance is reduced.

Variants

Piecewise-linear variants

Leaky ReLU 
Leaky ReLUs allow a small, positive gradient when the unit is not active, helping to mitigate the vanishing gradient problem.

Parametric ReLU 
Parametric ReLUs (PReLUs) take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural-network parameters.

Note that for a ≤ 1, this is equivalent to
 
and thus has a relation to "maxout" networks.

Other non-linear variants

Gaussian-error linear unit (GELU) 
GELU is a smooth approximation to the rectifier:

where Φ(x) is the cumulative distribution function of the standard normal distribution. 

This activation function is illustrated in the figure at the start of this article. It has a non-monotonic “bump” when x < 0 and serves as the default activation for models such as BERT.

SiLU 

The SiLU (sigmoid linear unit) or swish function is another smooth approximation, first coined in the GELU paper:

where  is the sigmoid function.

Softplus

A smooth approximation to the rectifier is the analytic function

which is called the softplus or SmoothReLU function. For large negative  it is roughly , so just above 0, while for large positive  it is roughly , so just above . 

A sharpness parameter  may be included:

The derivative of softplus is the logistic function.

The logistic sigmoid function is a smooth approximation of the derivative of the rectifier, the Heaviside step function.

The multivariable generalization of single-variable softplus is the LogSumExp with the first argument set to zero:
 
The LogSumExp function is
 
and its gradient is the softmax; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning.

ELU 
Exponential linear units try to make the mean activations closer to zero, which speeds up learning. It has been shown that ELUs can obtain higher classification accuracy than ReLUs.

In these formulas,  is a hyper-parameter to be tuned with the constraint .

The ELU can be viewed as a smoothed version of a shifted ReLU (SReLU), which has the form , given the same interpretation of .

Mish 
The mish function could also be used as a smooth approximation of the rectifier. It is defined as 

 

where  is the hyperbolic tangent, and  is the softplus function.

Mish is non-monotonic and self-gated. It was inspired by Swish, itself a variant of ReLU.

Squareplus 

Squareplus is the function

where  is a hyperparameter that determines the "size" of the curved region near . (For example, letting  yields ReLU, and letting  yields the metallic mean function.)
Squareplus shares many properties with softplus: It is monotonic, strictly positive, approaches 0 as , approaches the identity as , and is  smooth. However, squareplus can be computed using only algebraic functions, making it well-suited for settings where computational resources or instruction sets are limited. Additionally, squareplus requires no special consideration to ensure numerical stability when  is large.

See also
Softmax function
Sigmoid function
Tobit model
Layer (deep learning)

References

Artificial neural networks