Answers.Body: <p>For a discussion about the advantages of ReLU, see the <a href="https://proceedings.mlr.press/v15/glorot11a.html" rel="nofollow noreferrer">original paper by Glorot</a> (2011) &quot;Deep sparse rectifier neural networks&quot;.
  &quot;Efficient Backprop&quot; is a 1998 paper. At the time the use of rectifiers was uncommon and sigmoid was the standard choice of activation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T08:11:39.757Z
Answers.LastActivityDate: 2022-07-06T08:11:39.757Z
Answers.UserDisplayName: Martino
Body: <p>I was reading LeCun Efficient Backprop and the author repeated stressed the importance of average the input patterns at 0 and thus justified the usage of tanh sigmoid. But if tanh is good then how come ReLU is very popular in most NNs (which is even more odd when the authors didn't mention about ReLU at all)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-05T00:02:33.937Z
Comments.Text: Are you reading the original 1998 paper?  That would be my guess as to why ReLU isn't mentioned.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-06T08:43:38.267Z
Comments.Text: Does this answer your question? [Choices of activation functions](https://ai.stackexchange.com/questions/36189/choices-of-activation-functions)
Comments.UserDisplayName: Martino
Tags: <neural-networks><comparison><activation-functions><relu><tanh>
Title: Why and when do we use ReLU over tanh activation function?
UserDisplayName: Struggling_In_Final
ThreadText: # Why and when do we use ReLU over tanh activation function?
  
  (Asked by: username_0 on 2022-07-04)
  
  I was reading LeCun Efficient Backprop and the author repeated stressed the importance of average the input patterns at 0 and thus justified the usage of tanh sigmoid. But if tanh is good then how come ReLU is very popular in most NNs (which is even more odd when the authors didn't mention about ReLU at all)
  
  **Tags:** neural-networks, comparison, activation-functions, relu, tanh
  
  > **Comments:**
  > **username_1**: Are you reading the original 1998 paper?  That would be my guess as to why ReLU isn't mentioned.  
  > **username_2**: Does this answer your question? [Choices of activation functions](https://ai.stackexchange.com/questions/36189/choices-of-activation-functions)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-07-06. Score: 2
  
  For a discussion about the advantages of ReLU, see the [original paper by Glorot](https://proceedings.mlr.press/v15/glorot11a.html) (2011) "Deep sparse rectifier neural networks".
  "Efficient Backprop" is a 1998 paper. At the time the use of rectifiers was uncommon and sigmoid was the standard choice of activation.
  
  
  ---
  
  