Answers.Body: <p>The main characteristic of an activation function is to bring a non-linearity property into the neural network. For the hidden layer, there is no need for the function to be bounded. The last layer should use a function whose range corresponds to what you want.</p>
  <p>For regression, you usually re-scale your output data to <span class="math-container">$[-1,1]$</span> or <span class="math-container">$[0,1]$</span> and you use a <em>tanh</em> (hyperbolic tangent) or <em>sigmoid function</em> in the last layer</p>
  <p>For classification, you want to obtain probabilities: use a <em>softmax</em> function in the last layer.</p>
  <p>For the hidden layers some functions are better than others :</p>
  <ul>
  <li><p>The gradient should be fast to compute (from the perspective of your computer).</p>
  </li>
  <li><p>If you use too many hidden layers, you will have the vanishing gradient problem if the derivative of your activation is too close to zero. You need a large zone of the domain with a derivative not close to zero.</p>
  </li>
  </ul>
  <p>In practice, the ReLU function defined as <span class="math-container">$f(x)=\max(0, x)$</span> works very well and is very simple.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-06T08:09:08.847Z
Answers.LastActivityDate: 2020-11-09T10:08:03.390Z
Answers.LastEditDate: 2020-11-09T10:08:03.390Z
Answers.UserDisplayName: Adrien D
Body: <p>What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions?</p>
  
  <p>For example, which of the functions below can be used as an activation function?</p>
  
  <p><span class="math-container">$$f(x) = \frac{2}{\pi} \tan^{-1}(x)$$</span></p>
  
  <p>which looks like</p>
  
  <p><a href="https://i.sstatic.net/AXGlN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AXGlN.png" alt="enter image description here"></a></p>
  
  <p>or</p>
  
  <p><span class="math-container">$$f(x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{x} e^{-\frac{t^2}{2}} dt$$</span></p>
  
  <p>which looks </p>
  
  <p><a href="https://i.sstatic.net/GQE2B.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GQE2B.png" alt="enter image description here"></a></p>
  
Tags: <neural-networks><machine-learning><math><activation-functions>
Title: Which functions can be activation functions?
UserDisplayName: mBabaee
ThreadText: # Which functions can be activation functions?
  
  (Asked by: username_0 on 2018-06-05)
  
  What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions?
  
  For example, which of the functions below can be used as an activation function?
  
  $$f(x) = \frac{2}{\pi} \tan^{-1}(x)$$
  
  which looks like
  
  [![enter image description here](https://i.sstatic.net/AXGlN.png)](https://i.sstatic.net/AXGlN.png)
  
  or
  
  $$f(x) = \frac{1}{\sqrt{2\pi}} \int\_{-\infty}^{x} e^{-\frac{t^2}{2}} dt$$
  
  which looks
  
  [![enter image description here](https://i.sstatic.net/GQE2B.png)](https://i.sstatic.net/GQE2B.png)
  
  **Tags:** neural-networks, machine-learning, math, activation-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-06-06. Score: 3
  
  The main characteristic of an activation function is to bring a non-linearity property into the neural network. For the hidden layer, there is no need for the function to be bounded. The last layer should use a function whose range corresponds to what you want.
  
  For regression, you usually re-scale your output data to $[-1,1]$ or $[0,1]$ and you use a *tanh* (hyperbolic tangent) or *sigmoid function* in the last layer
  
  For classification, you want to obtain probabilities: use a *softmax* function in the last layer.
  
  For the hidden layers some functions are better than others :
  
  * The gradient should be fast to compute (from the perspective of your computer).
  * If you use too many hidden layers, you will have the vanishing gradient problem if the derivative of your activation is too close to zero. You need a large zone of the domain with a derivative not close to zero.
  
  In practice, the ReLU function defined as $f(x)=\max(0, x)$ works very well and is very simple.
  
  
  ---
  
  