Answers.Body: <p>The first part of this answer is regarding your concrete question and the second part summarizes things on activation functions in general because I believe that there are more important factors than what exactly the output spaces of the functions are, for the following reasons:</p>
  <p><strong>Should activation functions be zero-meaned:</strong></p>
  <p>I would say that in general, it is neither beneficial nor important for an activation function to be zero-mean.</p>
  <blockquote>
  <p>but what if the data at hand is 0.5 mean and we are not willing to zero mean the data? Would <code>tanh</code> still be the go-to option?</p>
  </blockquote>
  <p>IMO this intuition is a bit off because the activation functions are not applied to the data space directly, but to the latent spaces that are learnable:
  The matrix multiplication of the first layer will be applied prior to the activation function and it will change the topology of the data, hence also attributes like mean and variance. So your (let's say 2-dimensional) input data can have a mean of <code>[-1, -1]</code>, but if the matrix of the first layer learns something like a 180° rotation, the mean of the input to the activation function is now <code>[1, 1]</code>. Therefore, I would say that it is not so important for the choice of activation function what the mean of your data is.</p>
  <p>However, in certain cases, it may have some useful properties when the latent representations of your data are zero-meaned, but I would expect this to occur in shallow models rather than deep ones and I would say that this is something to find out empirically for the individual problem.</p>
  <p><strong>Disclaimer</strong> I don't think this question can be answered definitively, however, this is my understanding of the matter, happy to get comments and remarks on this.</p>
  <hr />
  <p><strong>Some facts regarding <code>sigmoid</code> <span class="math-container">$(\sigma)$</span> and <code>tanh</code></strong></p>
  <p><strong>Properties of the <code>sigmoid</code> <span class="math-container">$(\sigma)$</span></strong></p>
  <ul>
  <li><p>(-) The maximum of the derivative of the <span class="math-container">$\sigma$</span> is rather small (<span class="math-container">$\max \sigma'(x) = 0.25$</span>) which can slow down training and prevent deep models from training at all due to the vanishing gradient problem. (s. <a href="https://www.wolframalpha.com/input?i=derive+sigmoid" rel="nofollow noreferrer">derivative of <code>sigmoid</code></a> and <a href="https://www.wolframalpha.com/input?i=derive+tanh" rel="nofollow noreferrer">derivative of <code>tanh</code></a>)</p>
  </li>
  <li><p>(-) The sigmoid saturates towards <span class="math-container">$-\infty$</span> and <span class="math-container">$+\infty$</span>, thus the gradient becomes small when the pre-activation values are very positive or very negative values, this is also a cause for vanishing gradient and prevents the training of deep models.</p>
  </li>
  <li><p>(+) A very useful property is that <span class="math-container">$\sigma(x) \in (0, 1)$</span>. This is the reason why it is commonly used as an output activation for binary classification models. For the same reason, it is the function that is used in gates of recurrent models (s. <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRUCell.html" rel="nofollow noreferrer">GRU-cell</a>). So the sigmoid in this case can act as a continuous switch to determine the influence that, e.g. newly input features have on the hidden state of the model.</p>
  </li>
  </ul>
  <p><strong>Properties of the <code>tanh</code></strong></p>
  <ul>
  <li><p>(-) The <code>tanh</code> saturates towards <span class="math-container">$-\infty$</span> and <span class="math-container">$+\infty$</span> which poses similar drawbacks as for the sigmoid.</p>
  </li>
  <li><p>(o) The <code>tanh</code> can provide a larger gradient than the sigmoid (<span class="math-container">$\max \tanh'(x) = 1.0$</span>), although  there are other functions like the ReLU function that will output a large gradient more consistently.</p>
  </li>
  <li><p>(+) There probably exist some regression problems for which <code>tanh</code> is a good match as an output activation.</p>
  </li>
  <li><p>(+) The <span class="math-container">$\tanh(x) \in (-1, 1)$</span> this is probably one reason why it is commonly used as the activation function inside RNNs (vanilla, GRU, LSTM). This ensures that the hidden state will always be in this range and doesn't explode after several processing steps.</p>
  </li>
  </ul>
  <p><strong>What about the ReLU?</strong> The ReLU function is - to my knowledge - the most commonly used activation function for hidden layers (excluding RNNs). It consistently provides a gradient of one which leads to faster learning, especially for deep models. There are problems like dead neurons that occur when individual neurons produce only negative values (pre-activation) in which case the gradient and the output of that neuron will be zero til eternity. However, there are methods to reduce this problem, like controlling the bias, choosing a smaller learning rate, or switching to the Leaky-ReLu that can recover from 'dead' neurons.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-10T00:05:26.273Z
Answers.LastActivityDate: 2022-05-10T00:34:32.127Z
Answers.LastEditDate: 2022-05-10T00:34:32.127Z
Answers.UserDisplayName: Chillston
Body: <p>Conventionally, (although there are plenty of better options) it is being said that as the choice of activation function for hidden layers, <code>tanh</code> should be prefered over sigmoid because it has a zero mean but what if the data at hand is 0.5 mean and we are not willing to zero mean the data? Would <code>tanh</code> still be the go-to option?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-09T03:49:15.220Z
Comments.Text: i dont think it's important, sigmoid is just as fine as tanh
Comments.UserDisplayName: Dan D.
Tags: <neural-networks><deep-learning><activation-functions>
Title: Why is it important/beneficial for an activation function to be zero-meaned?
UserDisplayName: Onat Girit
ThreadText: # Why is it important/beneficial for an activation function to be zero-meaned?
  
  (Asked by: username_0 on 2022-05-06)
  
  Conventionally, (although there are plenty of better options) it is being said that as the choice of activation function for hidden layers, `tanh` should be prefered over sigmoid because it has a zero mean but what if the data at hand is 0.5 mean and we are not willing to zero mean the data? Would `tanh` still be the go-to option?
  
  **Tags:** neural-networks, deep-learning, activation-functions
  
  > **Comments:**
  > **username_1**: i dont think it's important, sigmoid is just as fine as tanh  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-05-10. Score: 1
  
  The first part of this answer is regarding your concrete question and the second part summarizes things on activation functions in general because I believe that there are more important factors than what exactly the output spaces of the functions are, for the following reasons:
  
  **Should activation functions be zero-meaned:**
  
  I would say that in general, it is neither beneficial nor important for an activation function to be zero-mean.
  
  > but what if the data at hand is 0.5 mean and we are not willing to zero mean the data? Would `tanh` still be the go-to option?
  
  IMO this intuition is a bit off because the activation functions are not applied to the data space directly, but to the latent spaces that are learnable:
  The matrix multiplication of the first layer will be applied prior to the activation function and it will change the topology of the data, hence also attributes like mean and variance. So your (let's say 2-dimensional) input data can have a mean of `[-1, -1]`, but if the matrix of the first layer learns something like a 180° rotation, the mean of the input to the activation function is now `[1, 1]`. Therefore, I would say that it is not so important for the choice of activation function what the mean of your data is.
  
  However, in certain cases, it may have some useful properties when the latent representations of your data are zero-meaned, but I would expect this to occur in shallow models rather than deep ones and I would say that this is something to find out empirically for the individual problem.
  
  **Disclaimer** I don't think this question can be answered definitively, however, this is my understanding of the matter, happy to get comments and remarks on this.
  
  ---
  
  **Some facts regarding `sigmoid` $(\sigma)$ and `tanh`**
  
  **Properties of the `sigmoid` $(\sigma)$**
  
  * (-) The maximum of the derivative of the $\sigma$ is rather small ($\max \sigma'(x) = 0.25$) which can slow down training and prevent deep models from training at all due to the vanishing gradient problem. (s. [derivative of `sigmoid`](https://www.wolframalpha.com/input?i=derive+sigmoid) and [derivative of `tanh`](https://www.wolframalpha.com/input?i=derive+tanh))
  * (-) The sigmoid saturates towards $-\infty$ and $+\infty$, thus the gradient becomes small when the pre-activation values are very positive or very negative values, this is also a cause for vanishing gradient and prevents the training of deep models.
  * (+) A very useful property is that $\sigma(x) \in (0, 1)$. This is the reason why it is commonly used as an output activation for binary classification models. For the same reason, it is the function that is used in gates of recurrent models (s. [GRU-cell](https://pytorch.org/docs/stable/generated/torch.nn.GRUCell.html)). So the sigmoid in this case can act as a continuous switch to determine the influence that, e.g. newly input features have on the hidden state of the model.
  
  **Properties of the `tanh`**
  
  * (-) The `tanh` saturates towards $-\infty$ and $+\infty$ which poses similar drawbacks as for the sigmoid.
  * (o) The `tanh` can provide a larger gradient than the sigmoid ($\max \tanh'(x) = 1.0$), although there are other functions like the ReLU function that will output a large gradient more consistently.
  * (+) There probably exist some regression problems for which `tanh` is a good match as an output activation.
  * (+) The $\tanh(x) \in (-1, 1)$ this is probably one reason why it is commonly used as the activation function inside RNNs (vanilla, GRU, LSTM). This ensures that the hidden state will always be in this range and doesn't explode after several processing steps.
  
  **What about the ReLU?** The ReLU function is - to my knowledge - the most commonly used activation function for hidden layers (excluding RNNs). It consistently provides a gradient of one which leads to faster learning, especially for deep models. There are problems like dead neurons that occur when individual neurons produce only negative values (pre-activation) in which case the gradient and the output of that neuron will be zero til eternity. However, there are methods to reduce this problem, like controlling the bias, choosing a smaller learning rate, or switching to the Leaky-ReLu that can recover from 'dead' neurons.
  
  
  ---
  
  