<p>It should be mentioned that RELU is the current activation function standard. But to answer your question:
The importance here is that it is very common to have normalized your data (e.g. using <a href="http://arxiv.org/pdf/1502.03167.pdf" rel="nofollow noreferrer">batch normalization</a>), then the data is centered around 0.</p>

<p>As @DuttaA commented, look at <a href="https://stats.stackexchange.com/questions/101560/tanh-activation-function-vs-sigmoid-activation-function">this answer</a> from Cross-Validated: </p>

<blockquote>
  <p>Since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that [output] values are in the range [0,1]. </p>
</blockquote>

<p>And</p>

<blockquote>
  <p>The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1] Avoiding bias in the gradients. This is explained very well in <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" rel="nofollow noreferrer">the paper</a>, and it is worth reading it to understand these issues.</p>
</blockquote>

CC BY-SA 3.0
2018-03-28T10:18:51.690Z
2018-03-28T10:25:21.237Z
2018-03-28T10:25:21.237Z
Andreas Storvik Strauman
<p>I have started reading <a href="https://books.google.com/books/about/Fundamentals_of_Deep_Learning.html?id=SL0BvgAACAAJ" rel="nofollow noreferrer">Fundamentals of Deep Learning by Nikhil Buduma</a> and I have a question regarding tanh neurons. In the book, it is stated:</p>

<blockquote>
  <p>"When S-shaped nonlinearities are used, the tanh neuron is often preferred over the sigmoid neuron because it is zero-centered."</p>
</blockquote>

<p>Can anyone explain me why exactly??</p>

<neural-networks><machine-learning>
S-shaped nonlinearities in tanh neurons
Gala
# S-shaped nonlinearities in tanh neurons

(Asked by: username_0 on 2018-03-26)

I have started reading [Fundamentals of Deep Learning by Nikhil Buduma](https://books.google.com/books/about/Fundamentals_of_Deep_Learning.html?id=SL0BvgAACAAJ) and I have a question regarding tanh neurons. In the book, it is stated:

> "When S-shaped nonlinearities are used, the tanh neuron is often preferred over the sigmoid neuron because it is zero-centered."

Can anyone explain me why exactly??

**Tags:** neural-networks, machine-learning


---

## 1 answers

### Answer by username_1 on 2018-03-28. Score: 1 | Accepted answer

It should be mentioned that RELU is the current activation function standard. But to answer your question:
The importance here is that it is very common to have normalized your data (e.g. using [batch normalization](http://arxiv.org/pdf/1502.03167.pdf)), then the data is centered around 0.

As @DuttaA commented, look at [this answer](https://stats.stackexchange.com/questions/101560/tanh-activation-function-vs-sigmoid-activation-function) from Cross-Validated:

> Since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that [output] values are in the range [0,1].

And

> The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1] Avoiding bias in the gradients. This is explained very well in [the paper](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), and it is worth reading it to understand these issues.


---

