<p>The main characteristic of an activation function is to bring a non-linearity property into the neural network. For the hidden layer, there is no need for the function to be bounded. The last layer should use a function whose range corresponds to what you want.</p>
<p>For regression, you usually re-scale your output data to <span class="math-container">$[-1,1]$</span> or <span class="math-container">$[0,1]$</span> and you use a <em>tanh</em> (hyperbolic tangent) or <em>sigmoid function</em> in the last layer</p>
<p>For classification, you want to obtain probabilities: use a <em>softmax</em> function in the last layer.</p>
<p>For the hidden layers some functions are better than others :</p>
<ul>
<li><p>The gradient should be fast to compute (from the perspective of your computer).</p>
</li>
<li><p>If you use too many hidden layers, you will have the vanishing gradient problem if the derivative of your activation is too close to zero. You need a large zone of the domain with a derivative not close to zero.</p>
</li>
</ul>
<p>In practice, the ReLU function defined as <span class="math-container">$f(x)=\max(0, x)$</span> works very well and is very simple.</p>

CC BY-SA 4.0
2018-06-06T08:09:08.847Z
2020-11-09T10:08:03.390Z
2020-11-09T10:08:03.390Z
Adrien D
<p>What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions?</p>

<p>For example, which of the functions below can be used as an activation function?</p>

<p><span class="math-container">$$f(x) = \frac{2}{\pi} \tan^{-1}(x)$$</span></p>

<p>which looks like</p>

<p><a href="https://i.sstatic.net/AXGlN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AXGlN.png" alt="enter image description here"></a></p>

<p>or</p>

<p><span class="math-container">$$f(x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{x} e^{-\frac{t^2}{2}} dt$$</span></p>

<p>which looks </p>

<p><a href="https://i.sstatic.net/GQE2B.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GQE2B.png" alt="enter image description here"></a></p>

<neural-networks><machine-learning><math><activation-functions>
Which functions can be activation functions?
mBabaee
# Which functions can be activation functions?

(Asked by: username_0 on 2018-06-05)

What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions?

For example, which of the functions below can be used as an activation function?

$$f(x) = \frac{2}{\pi} \tan^{-1}(x)$$

which looks like

[![enter image description here](https://i.sstatic.net/AXGlN.png)](https://i.sstatic.net/AXGlN.png)

or

$$f(x) = \frac{1}{\sqrt{2\pi}} \int\_{-\infty}^{x} e^{-\frac{t^2}{2}} dt$$

which looks

[![enter image description here](https://i.sstatic.net/GQE2B.png)](https://i.sstatic.net/GQE2B.png)

**Tags:** neural-networks, machine-learning, math, activation-functions


---

## 1 answers

### Answer by username_1 on 2018-06-06. Score: 3

The main characteristic of an activation function is to bring a non-linearity property into the neural network. For the hidden layer, there is no need for the function to be bounded. The last layer should use a function whose range corresponds to what you want.

For regression, you usually re-scale your output data to $[-1,1]$ or $[0,1]$ and you use a *tanh* (hyperbolic tangent) or *sigmoid function* in the last layer

For classification, you want to obtain probabilities: use a *softmax* function in the last layer.

For the hidden layers some functions are better than others :

* The gradient should be fast to compute (from the perspective of your computer).
* If you use too many hidden layers, you will have the vanishing gradient problem if the derivative of your activation is too close to zero. You need a large zone of the domain with a derivative not close to zero.

In practice, the ReLU function defined as $f(x)=\max(0, x)$ works very well and is very simple.


---

