Answers.Body: <p>In short, yes Leaky Relu helps in faster convergence if your output requires both positive and negative values. But the catch is that you need to tune the negative slope of Leaky Relu which is a hyperparameter to get better accuracy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-28T10:22:27.370Z
Answers.Text: By "tuning the negative slope" I mean that there is a parameter to Leaky Relu which has to be tuned for better results.
Answers.UserDisplayName: varsh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-27T09:50:19.457Z
Answers.LastActivityDate: 2019-01-27T09:50:19.457Z
Answers.UserDisplayName: varsh
Answers.Body: <p>Both the output and the the gradient at the negative part of leaky ReLU is 100 times lower than at the positive part. I doubt that they have any significant impact on training direction and/or on the final output of a trained model unless the model is severely underfitting.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-27T11:38:12.923Z
Answers.LastActivityDate: 2019-01-27T11:38:12.923Z
Answers.UserDisplayName: Kit.
Body: <p>In an MLP with ReLU activation functions after each hidden layer (except the final),</p>
  
  <p>Let's say the final layer should output positive and negative values.</p>
  
  <p>With ReLU intermediary activations, this is still possible because the final layer, despite taking in positive inputs only, can combine them to be negative.</p>
  
  <p>However, would using leaky ReLU allow faster convergence? Because you can pass in negative values as input to the final layer instead of waiting till the final layer to make things negative</p>
  
Tags: <neural-networks>
Title: Does leaky relu help learning if the final output needs negative values?
UserDisplayName: user3180
ThreadText: # Does leaky relu help learning if the final output needs negative values?
  
  (Asked by: username_0 on 2019-01-27)
  
  In an MLP with ReLU activation functions after each hidden layer (except the final),
  
  Let's say the final layer should output positive and negative values.
  
  With ReLU intermediary activations, this is still possible because the final layer, despite taking in positive inputs only, can combine them to be negative.
  
  However, would using leaky ReLU allow faster convergence? Because you can pass in negative values as input to the final layer instead of waiting till the final layer to make things negative
  
  **Tags:** neural-networks
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-01-27. Score: 2
  
  In short, yes Leaky Relu helps in faster convergence if your output requires both positive and negative values. But the catch is that you need to tune the negative slope of Leaky Relu which is a hyperparameter to get better accuracy.
  
  > **Comments:**
  > **username_1**: By "tuning the negative slope" I mean that there is a parameter to Leaky Relu which has to be tuned for better results.  
  
  ---
  
  ### Answer by username_2 on 2019-01-27. Score: 0
  
  Both the output and the the gradient at the negative part of leaky ReLU is 100 times lower than at the positive part. I doubt that they have any significant impact on training direction and/or on the final output of a trained model unless the model is severely underfitting.
  
  
  ---
  
  