Answers.Body: <p>Combining ReLU, the hyper-parameterized<sup>1</sup> leaky variant, and variant with dynamic parametrization during learning confuses two distinct things:</p>
  
  <ul>
  <li>The comparison between ReLU with the leaky variant is closely related to whether there is a need, in the particular ML case at hand, to avoid saturation &mdash; Saturation is thee loss of signal to either zero gradient<sup>2</sup> or the dominance of chaotic noise arising from digital rounding<sup>3</sup>.</li>
  <li>The comparison between training-dynamic activation (called <em>parametric</em> in the literature) and training-static activation must be based on whether the non-linear or non-smooth characteristics of activation have any value related to the rate of convergence<sup>4</sup>.</li>
  </ul>
  
  <p>The reason ReLU is never parametric is that to make it so would be redundant.  In the negative domain, it is the constant zero.  In the non-negative domain, its derivative is constant.  Since the activation input vector is already attenuated with a vector-matrix product (where the matrix, cube, or hyper-cube contains the attenuation parameters) there is no useful purpose in adding a parameter to vary the constant derivative for the non-negative domain.</p>
  
  <p>When there is curvature in the activation, it is no longer true that all the coefficients of activation are redundant as parameters.  Their values may considerably alter the training process and thus the speed and reliability of convergence.</p>
  
  <p>For substantially deep networks, the redundancy reemerges, and there is evidence of this, both in theory and practice in the literature.</p>
  
  <ul>
  <li>In algebraic terms, the disparity between ReLU and parametrically dynamic activations derived from it approaches zero as the depth (in number of layers) approaches infinity.</li>
  <li>In descriptive terms, ReLU can accurately approximate functions with curvature<sup>5</sup> if given a sufficient number of layers to do so.</li>
  </ul>
  
  <p>That is why the ELU variety, which is advantageous for averting the saturation issues mentioned above for shallower networks is not used for deeper ones.</p>
  
  <p>So one must decided two things.</p>
  
  <ul>
  <li>Whether parametric activation is helpful is often based on experimentation with several samples from a statistical population.  But there is no need to experiment at all with it if the layer depth is high.</li>
  <li>Whether the leaky variant is of value has much to do with the numerical ranges encountered during back propagation.  If the gradient becomes vanishingly small during back propagation at any point during training, a constant portion of the activation curve may be problematic.  In such a scase one of the smooth functions or leaky RelU with it's two non-zero slopes may provide adequate solution.</li>
  </ul>
  
  <p>In summary, the choice is never a choice of convenience.</p>
  
  <hr>
  
  <p><strong>Footnotes</strong></p>
  
  <p>[1] Hyper-parameters are parameters that affect the signaling through the layer that are not part of the attenuation of inputs for that layer.  The attenuation weights are parameters.  Any other parametrization is in the set of hyper-parameters.  This may include learning rate, dampening of high frequencies in the back propagation, and a wide variety of other learning controls that are set for the entire layer, if not the entire network.</p>
  
  <p>[2] If the gradient is zero, then there cannot be any intelligent adjustment of the parameters because the direction of the adjustment is unknown, and its magnitude must be zero.  Learning stops.</p>
  
  <p>[3] If chaotic noise, which can arise as the CPU rounds extremely small values to their closest digital representation, dominates the correction signal that is intended to propagate back to the layers, then the correction becomes nonsense and learning stops.</p>
  
  <p>[4] Rate of convergence is a measure of the speed (either relative to microseconds or relative to the iteration index of the algorithm) in which the result of learning (system behavior) approaches what is considered good enough.  That's usually some specified proximity to some formal acceptance criteria for the convergence (learning).</p>
  
  <p>[5] Functions with curvature are ones that are not visualized as straight or flat.  A parabola has curvature.  A straight line does not.  The surface of an egg has curvature.  A perfect flat plane does not.  Mathematically, if any of the elements of the Hessian of the function is non-zero, the function has curvature.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-26T20:18:20.967Z
Answers.Text: What do you mean by _The choice is never a choice of convenience_?
Answers.UserDisplayName: gvgramazio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-27T06:46:10.003Z
Answers.Text: @gvgramazio, You had written, "more convenient to use ReLU," in your question. I was indicating that convenience is not the basis on which the choice is made. Perhaps it comes off as too harsh? I didn't intend to be. The sentences above that sentence in my answer were intended to provide the more useful criteria upon which you might base your decision when choosing activation functions.
Answers.UserDisplayName: Douglas Daseeco
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-27T15:40:07.827Z
Answers.Text: Don't worry about being too harsh, not a problem at all for me. I think that is more a problem of language (I'm not a native English speaker).
Answers.UserDisplayName: gvgramazio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-27T15:50:41.700Z
Answers.Text: The fact is that, from what I've understood, you explain well in which case I should prefer one variant with respect to the other ones. What I still don't understand is when I should prefer the classical one. e.g. the leaky variant may provide an adequate solution to vanishing gradient but if it hasn't any drawback I could always choose the leaky variant with respect to the ReLU.
Answers.UserDisplayName: gvgramazio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-25T00:34:55.660Z
Answers.LastActivityDate: 2018-08-05T04:22:34.303Z
Answers.LastEditDate: 2018-08-05T04:22:34.303Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of the output for negative inputs is a learnable parameter while in the Leaky ReLU it's a hyperparameter.</p>
  
  <p>However, I'm not able to tell if there are cases where is more convenient to use ReLU instead of Leaky ReLU or Parametric ReLU.</p>
  
Tags: <neural-networks><activation-functions><relu>
Title: What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?
UserDisplayName: gvgramazio
ThreadText: # What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?
  
  (Asked by: username_0 on 2018-07-24)
  
  I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of the output for negative inputs is a learnable parameter while in the Leaky ReLU it's a hyperparameter.
  
  However, I'm not able to tell if there are cases where is more convenient to use ReLU instead of Leaky ReLU or Parametric ReLU.
  
  **Tags:** neural-networks, activation-functions, relu
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-07-25. Score: 14
  
  Combining ReLU, the hyper-parameterized1 leaky variant, and variant with dynamic parametrization during learning confuses two distinct things:
  
  * The comparison between ReLU with the leaky variant is closely related to whether there is a need, in the particular ML case at hand, to avoid saturation — Saturation is thee loss of signal to either zero gradient2 or the dominance of chaotic noise arising from digital rounding3.
  * The comparison between training-dynamic activation (called *parametric* in the literature) and training-static activation must be based on whether the non-linear or non-smooth characteristics of activation have any value related to the rate of convergence4.
  
  The reason ReLU is never parametric is that to make it so would be redundant. In the negative domain, it is the constant zero. In the non-negative domain, its derivative is constant. Since the activation input vector is already attenuated with a vector-matrix product (where the matrix, cube, or hyper-cube contains the attenuation parameters) there is no useful purpose in adding a parameter to vary the constant derivative for the non-negative domain.
  
  When there is curvature in the activation, it is no longer true that all the coefficients of activation are redundant as parameters. Their values may considerably alter the training process and thus the speed and reliability of convergence.
  
  For substantially deep networks, the redundancy reemerges, and there is evidence of this, both in theory and practice in the literature.
  
  * In algebraic terms, the disparity between ReLU and parametrically dynamic activations derived from it approaches zero as the depth (in number of layers) approaches infinity.
  * In descriptive terms, ReLU can accurately approximate functions with curvature5 if given a sufficient number of layers to do so.
  
  That is why the ELU variety, which is advantageous for averting the saturation issues mentioned above for shallower networks is not used for deeper ones.
  
  So one must decided two things.
  
  * Whether parametric activation is helpful is often based on experimentation with several samples from a statistical population. But there is no need to experiment at all with it if the layer depth is high.
  * Whether the leaky variant is of value has much to do with the numerical ranges encountered during back propagation. If the gradient becomes vanishingly small during back propagation at any point during training, a constant portion of the activation curve may be problematic. In such a scase one of the smooth functions or leaky RelU with it's two non-zero slopes may provide adequate solution.
  
  In summary, the choice is never a choice of convenience.
  
  ---
  
  **Footnotes**
  
  [1] Hyper-parameters are parameters that affect the signaling through the layer that are not part of the attenuation of inputs for that layer. The attenuation weights are parameters. Any other parametrization is in the set of hyper-parameters. This may include learning rate, dampening of high frequencies in the back propagation, and a wide variety of other learning controls that are set for the entire layer, if not the entire network.
  
  [2] If the gradient is zero, then there cannot be any intelligent adjustment of the parameters because the direction of the adjustment is unknown, and its magnitude must be zero. Learning stops.
  
  [3] If chaotic noise, which can arise as the CPU rounds extremely small values to their closest digital representation, dominates the correction signal that is intended to propagate back to the layers, then the correction becomes nonsense and learning stops.
  
  [4] Rate of convergence is a measure of the speed (either relative to microseconds or relative to the iteration index of the algorithm) in which the result of learning (system behavior) approaches what is considered good enough. That's usually some specified proximity to some formal acceptance criteria for the convergence (learning).
  
  [5] Functions with curvature are ones that are not visualized as straight or flat. A parabola has curvature. A straight line does not. The surface of an egg has curvature. A perfect flat plane does not. Mathematically, if any of the elements of the Hessian of the function is non-zero, the function has curvature.
  
  > **Comments:**
  > **username_0**: What do you mean by _The choice is never a choice of convenience_?  
  > **username_1**: @username_0, You had written, "more convenient to use ReLU," in your question. I was indicating that convenience is not the basis on which the choice is made. Perhaps it comes off as too harsh? I didn't intend to be. The sentences above that sentence in my answer were intended to provide the more useful criteria upon which you might base your decision when choosing activation functions.  
  > **username_0**: Don't worry about being too harsh, not a problem at all for me. I think that is more a problem of language (I'm not a native English speaker).  
  > **username_0**: The fact is that, from what I've understood, you explain well in which case I should prefer one variant with respect to the other ones. What I still don't understand is when I should prefer the classical one. e.g. the leaky variant may provide an adequate solution to vanishing gradient but if it hasn't any drawback I could always choose the leaky variant with respect to the ReLU.  
  
  ---
  
  