Answers.Body: <p>***Take my answer as a side note to that given by cantordust:</p>
  
  <p>If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems.
  Thus, by testing activation functions on a few different problems, one can often infer how well (or badly) it will perform on most problems.
  The following video shows how different activation functions perform in different problems:</p>
  
  <p><a href="https://www.youtube.com/watch?v=Hb3vIYUQ_I8" rel="nofollow noreferrer">https://www.youtube.com/watch?v=Hb3vIYUQ_I8</a></p>
  
  <p>One can verify that an activation function usually perform well in all cases, or the other way around: it does it poorly in all cases.
  As cantordust says, I would recommend always starting with leaky ReLU: it is simple, efficient, and generally produces nice results in a wide variety of problems.
  It also evades the dying ReLU problem, and does not suffer from the vanishing gradient problem.
  The only thing to keep in mind is the exploding gradient problem if the neural network is too deep, or if it is a recurrent neural network, which are essentially the same concept.</p>
  
  <p>The video shows that other activation functions worth trying (in addition to leaky ReLU) are Gaussian, Sinusoid, or Tanh.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-24T23:05:39.870Z
Answers.Text: May I ask, what do you mean by  "It also evades the dying ReLU problem, and does not suffer from the vanishing gradient problem."?
Answers.UserDisplayName: Verónica Rmz.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-25T16:39:51.740Z
Answers.Text: leaky ReLUs take values different than zero for negative inputs. The derivative is different than zero for those values. Then, your gradient will not "vanish", and your gradient descent/ascent algorithm will be able to change the weight matrix in zones where ReLUs would be "dead" (because their derivative would be zero).
Answers.UserDisplayName: pbp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-11T12:25:13.987Z
Answers.LastActivityDate: 2020-06-11T17:41:09.913Z
Answers.LastEditDate: 2020-06-11T17:41:09.913Z
Answers.UserDisplayName: pbp
Answers.Body: <p>I don't know what kind of neural networks you are working on. But one should also consider tanh activation functions when dealing with recurrent neural network. The why is to avoid exploding gradient issues since the tanh function is bounded at the difference of the RELU function for instance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-09T16:11:51.257Z
Answers.Text: In the question I stated that I use tanh and sigmoid, not only ReLU. Also, to keep it simple I'm referring in general to classical hidden fully connected layers. If you think that the fact that we are dealing with a recurrent neural network is significant for the choice of the activation function please state the reason for that. The exploding/vanishing phenomena could happens in non recurrent neural network as well.
Answers.UserDisplayName: gvgramazio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-09T13:51:27.067Z
Answers.LastActivityDate: 2018-07-09T13:51:27.067Z
Answers.UserDisplayName: Alexis
Answers.Body: <p>It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU).</p>
  <p>You may want to look at <a href="https://arxiv.org/abs/1511.07289" rel="nofollow noreferrer"><strong>ELU</strong></a> (exponential linear units) and <a href="https://arxiv.org/abs/1706.02515" rel="nofollow noreferrer"><strong>SELU</strong></a> (self-normalising version of ELU). Under some mild assumptions, the latter has the nice property of self-normalisation, which mitigates the problem of vanishing and exploding gradients. In addition, they <em>propagate</em> normalisation - i.e., they guarantee that the input to the next layer will have zero mean and unit variance.</p>
  <p>However, it would be incredibly difficult to recommend an activation function (AF) that works for all use cases, although I believe that SELU was designed so that it would do the right thing with pretty much any input.</p>
  <p>There are many considerations - how difficult it is to compute the derivative (if it is differentiable at all!), how quickly a NN with your chosen AF converges, how smooth it is, whether it satisfies the conditions of the <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">universal approximation theorem</a>, whether it preserves normalisation, and so on. You may or may not care about some or any of those.</p>
  <p>The bottom line is that there is no universal rule for choosing an activation function for hidden layers. Personally, I like to use sigmoids (especially <em>tanh</em>) because they are nicely bounded and very fast to compute, but most importantly because they <em>work for my use cases</em>. Others <a href="https://www.coursera.org/lecture/ai/how-to-choose-the-correct-activation-function-foyh8" rel="nofollow noreferrer">recommend</a> <em>leaky ReLU</em> for the input and hidden layers as a go-to function if your NN fails to learn. You can even <a href="https://arxiv.org/pdf/1703.07122.pdf" rel="nofollow noreferrer">mix and match</a> activation functions to evolve NNs for <a href="http://picbreeder.org/" rel="nofollow noreferrer">fancy applications</a>.</p>
  <p>At the end of the day, you are probably going to get as many opinions as there are people about the right choice of activation function, so the short answer should probably be: start with the AF of the day (leaky ReLU / SELU?) and work your way through other AFs in order of decreasing popularity if your NN struggles to learn anything.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-31T03:56:34.560Z
Answers.Text: If my input is a length 4 vector containing only positive number, and my output is also a length 4 vector containing only positive number. Then, ReLU, leaky ReLU would always be a good choice, am I correct? not the tanh, sigmoid, softmax, etc.
Answers.UserDisplayName: FantasticAI
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-09T01:44:31.187Z
Answers.LastActivityDate: 2020-12-09T21:07:24.677Z
Answers.LastEditDate: 2020-12-09T21:07:24.677Z
Answers.UserDisplayName: cantordust
Body: <p>I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the <em>sigmoid</em> function when I'm dealing with probabilities, a <em>ReLU</em> when I'm dealing with positive values, and a linear function when I'm dealing with general values.</p>
  <p>In hidden layers, I use a <em>leaky ReLU</em> to avoid dead neurons instead of the ReLU, and the <em>tanh</em> instead of the <em>sigmoid</em>. Of course, I don't use a linear function in hidden units.</p>
  <p>However, the choice for them in the hidden layer is mostly due to trial and error.</p>
  <p>Is there any rule of thumb of which activation function is likely to work well in some situations?</p>
  <p>Take the term <em>situations</em> as general as possible: it could be referring to the depth of the layer, to the depth of the NN, to the number of neurons for that layer, to the optimizer that we chose, to the number of input features of that layer, to the application of this NN, etc.</p>
  <p>The more activation functions I discover the more I'm confused in the choice of the function to use in hidden layers. I don't think that flipping a coin is a good way of choosing an activation function.</p>
  
Tags: <neural-networks><deep-learning><activation-functions><hyperparameter-optimization><hyper-parameters>
Title: How to choose an activation function for the hidden layers?
UserDisplayName: gvgramazio
ThreadText: # How to choose an activation function for the hidden layers?
  
  (Asked by: username_0 on 2018-07-09)
  
  I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the *sigmoid* function when I'm dealing with probabilities, a *ReLU* when I'm dealing with positive values, and a linear function when I'm dealing with general values.
  
  In hidden layers, I use a *leaky ReLU* to avoid dead neurons instead of the ReLU, and the *tanh* instead of the *sigmoid*. Of course, I don't use a linear function in hidden units.
  
  However, the choice for them in the hidden layer is mostly due to trial and error.
  
  Is there any rule of thumb of which activation function is likely to work well in some situations?
  
  Take the term *situations* as general as possible: it could be referring to the depth of the layer, to the depth of the NN, to the number of neurons for that layer, to the optimizer that we chose, to the number of input features of that layer, to the application of this NN, etc.
  
  The more activation functions I discover the more I'm confused in the choice of the function to use in hidden layers. I don't think that flipping a coin is a good way of choosing an activation function.
  
  **Tags:** neural-networks, deep-learning, activation-functions, hyperparameter-optimization, hyper-parameters
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2018-07-09. Score: 16 | Accepted answer
  
  It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU).
  
  You may want to look at [**ELU**](https://arxiv.org/abs/1511.07289) (exponential linear units) and [**SELU**](https://arxiv.org/abs/1706.02515) (self-normalising version of ELU). Under some mild assumptions, the latter has the nice property of self-normalisation, which mitigates the problem of vanishing and exploding gradients. In addition, they *propagate* normalisation - i.e., they guarantee that the input to the next layer will have zero mean and unit variance.
  
  However, it would be incredibly difficult to recommend an activation function (AF) that works for all use cases, although I believe that SELU was designed so that it would do the right thing with pretty much any input.
  
  There are many considerations - how difficult it is to compute the derivative (if it is differentiable at all!), how quickly a NN with your chosen AF converges, how smooth it is, whether it satisfies the conditions of the [universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem), whether it preserves normalisation, and so on. You may or may not care about some or any of those.
  
  The bottom line is that there is no universal rule for choosing an activation function for hidden layers. Personally, I like to use sigmoids (especially *tanh*) because they are nicely bounded and very fast to compute, but most importantly because they *work for my use cases*. Others [recommend](https://www.coursera.org/lecture/ai/how-to-choose-the-correct-activation-function-foyh8) *leaky ReLU* for the input and hidden layers as a go-to function if your NN fails to learn. You can even [mix and match](https://arxiv.org/pdf/1703.07122.pdf) activation functions to evolve NNs for [fancy applications](http://picbreeder.org/).
  
  At the end of the day, you are probably going to get as many opinions as there are people about the right choice of activation function, so the short answer should probably be: start with the AF of the day (leaky ReLU / SELU?) and work your way through other AFs in order of decreasing popularity if your NN struggles to learn anything.
  
  > **Comments:**
  > **username_5**: If my input is a length 4 vector containing only positive number, and my output is also a length 4 vector containing only positive number. Then, ReLU, leaky ReLU would always be a good choice, am I correct? not the tanh, sigmoid, softmax, etc.  
  
  ---
  
  ### Answer by username_1 on 2020-06-11. Score: 3
  
  \*\*\*Take my answer as a side note to that given by username_4:
  
  If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems.
  Thus, by testing activation functions on a few different problems, one can often infer how well (or badly) it will perform on most problems.
  The following video shows how different activation functions perform in different problems:
  
  <https://www.youtube.com/watch?v=Hb3vIYUQ_I8>
  
  One can verify that an activation function usually perform well in all cases, or the other way around: it does it poorly in all cases.
  As username_4 says, I would recommend always starting with leaky ReLU: it is simple, efficient, and generally produces nice results in a wide variety of problems.
  It also evades the dying ReLU problem, and does not suffer from the vanishing gradient problem.
  The only thing to keep in mind is the exploding gradient problem if the neural network is too deep, or if it is a recurrent neural network, which are essentially the same concept.
  
  The video shows that other activation functions worth trying (in addition to leaky ReLU) are Gaussian, Sinusoid, or Tanh.
  
  > **Comments:**
  > **username_2**: May I ask, what do you mean by  "It also evades the dying ReLU problem, and does not suffer from the vanishing gradient problem."?  
  > **username_1**: leaky ReLUs take values different than zero for negative inputs. The derivative is different than zero for those values. Then, your gradient will not "vanish", and your gradient descent/ascent algorithm will be able to change the weight matrix in zones where ReLUs would be "dead" (because their derivative would be zero).  
  
  ---
  
  ### Answer by username_3 on 2018-07-09. Score: 0
  
  I don't know what kind of neural networks you are working on. But one should also consider tanh activation functions when dealing with recurrent neural network. The why is to avoid exploding gradient issues since the tanh function is bounded at the difference of the RELU function for instance.
  
  > **Comments:**
  > **username_0**: In the question I stated that I use tanh and sigmoid, not only ReLU. Also, to keep it simple I'm referring in general to classical hidden fully connected layers. If you think that the fact that we are dealing with a recurrent neural network is significant for the choice of the activation function please state the reason for that. The exploding/vanishing phenomena could happens in non recurrent neural network as well.  
  
  ---
  
  