Answers.Body: <p>Your understanding or Leaky ReLU is correct, and, yes, it has been proposed to mitigate the <em>dying neurons issue</em> in ReLU: when these are negative, they got zeroed.</p>
  <p>Regarding the answer of @Regresslt:</p>
  <ul>
  <li>I would not say that leaky relu slows done training due to its computation overhead, since it's quite simple to implement. Instead, ELU (exponential linear unit) is quite expensive and that can increase training time.</li>
  <li>Also it's true that the <em>slope</em> parameter (usually called <span class="math-container">$\alpha$</span>), if mistuned can lead to worst performance, but one can try the PReLU that learns it.</li>
  </ul>
  <blockquote>
  <p>Does the additional sparsity outweigh the value gained from negative local gradients?</p>
  </blockquote>
  <p>That's an interesting. I mean the dying ReLU phenomenon is usually observed on a fraction of the units, not the totality. In my personal experience I've noticed (dense) layers with ReLU to be sparse as much as <span class="math-container">$50$</span> to <span class="math-container">$90\%$</span>. In general this causes an under utilization of the capacity of the model, which may lead to under-fitting. But from another perspective can be seen as <em>implicit</em> <span class="math-container">$l_1$</span>-regularization applied on each layer with ReLU, that is optimized along the model which may lead to better generalization performance: simply because the model learned to achieve the right capacity by itself. Also, if such large sparsity is achieved the use of Dropout on top can be totally non-beneficial, because it would drop also the remaining &quot;active&quot; units/neurons.</p>
  <blockquote>
  <p>Is tanh just too sparse?</p>
  </blockquote>
  <p>First, <span class="math-container">$\tanh$</span> is not sparse since the output values can be either <span class="math-container">$-1$</span> or <span class="math-container">$1$</span> when <em>saturation</em> is reached, which is its main issue that shares with <em>sigmoid</em> (that instead can lead to sparsity.)  When tanh or sigmoid saturate you get vanishing gradients, and the training stops. Instead, the sparsity induced by ReLU does not reduce the gradient, since it will only be multiplied by some zero elements (corresponding to the died units.): the gradient magnitude depends on the overall output of the layer. In practice (but I'd say it's hard to happen), only if <em>all</em> units of <em>all</em> layers are died your network won't learn anymore, and the gradient would vanish because each layers would only output zeroes.</p>
  <p>To conclude, ReLU is usually the first activation to try - it seems to always work sufficiently well - but remember also to pair the activation fn with the right choice of <em>weight initialization</em>, in order to prevent issues (dying units, saturation, etc), speed-up convergence, and even increase performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-25T10:13:39.693Z
Answers.LastActivityDate: 2023-05-25T10:13:39.693Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive.</p>
  <p>With just this info to go off of, it would seem that the leaky ReLU is just an overall improvement to the standard ReLU, yet ReLU still seems to be the gold standard of activation functions. Why is that?</p>
  <p>Does the additional sparsity outweigh the value gained from negative local gradients? If so, is tanh just too sparse?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-12T17:53:16.960Z
Comments.Text: Can I ask what you mean by sparsity?
Comments.UserDisplayName: JobHunter69
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-14T19:31:28.257Z
Comments.Text: @JobHunter69 A sparse activation function is more strict when it comes to letting non-zero activations through than a non-sparse function. This is beneficial b/c it leads to a more generalizable model that "focuses" on only the most distinguishing features of the data. Read Deep Sparse Rectifier Neural Networks by Bengio et al. for a more thorough explanation of the topic.
Comments.UserDisplayName: John Brown
Tags: <machine-learning><deep-learning><backpropagation><gradient-descent><activation-functions>
Title: Why use ReLU over Leaky ReLU?
UserDisplayName: John Brown
ThreadText: # Why use ReLU over Leaky ReLU?
  
  (Asked by: username_0 on 2023-05-24)
  
  From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive.
  
  With just this info to go off of, it would seem that the leaky ReLU is just an overall improvement to the standard ReLU, yet ReLU still seems to be the gold standard of activation functions. Why is that?
  
  Does the additional sparsity outweigh the value gained from negative local gradients? If so, is tanh just too sparse?
  
  **Tags:** machine-learning, deep-learning, backpropagation, gradient-descent, activation-functions
  > **Comments:**
  > **username_1**: Can I ask what you mean by sparsity?  
  > **username_0**: @JobHunter69 A sparse activation function is more strict when it comes to letting non-zero activations through than a non-sparse function. This is beneficial b/c it leads to a more generalizable model that "focuses" on only the most distinguishing features of the data. Read Deep Sparse Rectifier Neural Networks by Bengio et al. for a more thorough explanation of the topic.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-25. Score: 5 | Accepted answer
  
  Your understanding or Leaky ReLU is correct, and, yes, it has been proposed to mitigate the *dying neurons issue* in ReLU: when these are negative, they got zeroed.
  
  Regarding the answer of @Regresslt:
  
  * I would not say that leaky relu slows done training due to its computation overhead, since it's quite simple to implement. Instead, ELU (exponential linear unit) is quite expensive and that can increase training time.
  * Also it's true that the *slope* parameter (usually called $\alpha$), if mistuned can lead to worst performance, but one can try the PReLU that learns it.
  
  > Does the additional sparsity outweigh the value gained from negative local gradients?
  
  That's an interesting. I mean the dying ReLU phenomenon is usually observed on a fraction of the units, not the totality. In my personal experience I've noticed (dense) layers with ReLU to be sparse as much as $50$ to $90\%$. In general this causes an under utilization of the capacity of the model, which may lead to under-fitting. But from another perspective can be seen as *implicit* $l\_1$-regularization applied on each layer with ReLU, that is optimized along the model which may lead to better generalization performance: simply because the model learned to achieve the right capacity by itself. Also, if such large sparsity is achieved the use of Dropout on top can be totally non-beneficial, because it would drop also the remaining "active" units/neurons.
  
  > Is tanh just too sparse?
  
  First, $\tanh$ is not sparse since the output values can be either $-1$ or $1$ when *saturation* is reached, which is its main issue that shares with *sigmoid* (that instead can lead to sparsity.) When tanh or sigmoid saturate you get vanishing gradients, and the training stops. Instead, the sparsity induced by ReLU does not reduce the gradient, since it will only be multiplied by some zero elements (corresponding to the died units.): the gradient magnitude depends on the overall output of the layer. In practice (but I'd say it's hard to happen), only if *all* units of *all* layers are died your network won't learn anymore, and the gradient would vanish because each layers would only output zeroes.
  
  To conclude, ReLU is usually the first activation to try - it seems to always work sufficiently well - but remember also to pair the activation fn with the right choice of *weight initialization*, in order to prevent issues (dying units, saturation, etc), speed-up convergence, and even increase performance.
  
  
  ---
  
  