Answers.Body: <p>It should be mentioned that RELU is the current activation function standard. But to answer your question:
  The importance here is that it is very common to have normalized your data (e.g. using <a href="http://arxiv.org/pdf/1502.03167.pdf" rel="nofollow noreferrer">batch normalization</a>), then the data is centered around 0.</p>
  
  <p>As @DuttaA commented, look at <a href="https://stats.stackexchange.com/questions/101560/tanh-activation-function-vs-sigmoid-activation-function">this answer</a> from Cross-Validated: </p>
  
  <blockquote>
    <p>Since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that [output] values are in the range [0,1]. </p>
  </blockquote>
  
  <p>And</p>
  
  <blockquote>
    <p>The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1] Avoiding bias in the gradients. This is explained very well in <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" rel="nofollow noreferrer">the paper</a>, and it is worth reading it to understand these issues.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-28T10:18:51.690Z
Answers.LastActivityDate: 2018-03-28T10:25:21.237Z
Answers.LastEditDate: 2018-03-28T10:25:21.237Z
Answers.UserDisplayName: Andreas Storvik Strauman
Body: <p>I have started reading <a href="https://books.google.com/books/about/Fundamentals_of_Deep_Learning.html?id=SL0BvgAACAAJ" rel="nofollow noreferrer">Fundamentals of Deep Learning by Nikhil Buduma</a> and I have a question regarding tanh neurons. In the book, it is stated:</p>
  
  <blockquote>
    <p>"When S-shaped nonlinearities are used, the tanh neuron is often preferred over the sigmoid neuron because it is zero-centered."</p>
  </blockquote>
  
  <p>Can anyone explain me why exactly??</p>
  
Tags: <neural-networks><machine-learning>
Title: S-shaped nonlinearities in tanh neurons
UserDisplayName: Gala
ThreadText: # S-shaped nonlinearities in tanh neurons
  
  (Asked by: username_0 on 2018-03-26)
  
  I have started reading [Fundamentals of Deep Learning by Nikhil Buduma](https://books.google.com/books/about/Fundamentals_of_Deep_Learning.html?id=SL0BvgAACAAJ) and I have a question regarding tanh neurons. In the book, it is stated:
  
  > "When S-shaped nonlinearities are used, the tanh neuron is often preferred over the sigmoid neuron because it is zero-centered."
  
  Can anyone explain me why exactly??
  
  **Tags:** neural-networks, machine-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-03-28. Score: 1 | Accepted answer
  
  It should be mentioned that RELU is the current activation function standard. But to answer your question:
  The importance here is that it is very common to have normalized your data (e.g. using [batch normalization](http://arxiv.org/pdf/1502.03167.pdf)), then the data is centered around 0.
  
  As @DuttaA commented, look at [this answer](https://stats.stackexchange.com/questions/101560/tanh-activation-function-vs-sigmoid-activation-function) from Cross-Validated:
  
  > Since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that [output] values are in the range [0,1].
  
  And
  
  > The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1] Avoiding bias in the gradients. This is explained very well in [the paper](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), and it is worth reading it to understand these issues.
  
  
  ---
  
  