Answers.Body: <p>Yes, if the activation function of the network is <strong>not zero centered</strong>, <span class="math-container">$y = f(x^{T}w)$</span> is always <em>positive</em> or always <em>negative</em>. Thus, the output of a layer is always being moved to either the positive values or the negative values. As a result, the <strong>weight vector</strong> needs <strong>more updates</strong> to be trained properly, and the number of <strong>epochs</strong> needed for the network to get trained also <strong>increases</strong>. This is why the <strong>zero centered property is important</strong>, though it is <strong>NOT</strong> necessary.</p>
  <p><strong>Zero-centered activation functions</strong> ensure that the <strong>mean activation value</strong> is around <strong>zero</strong>. This property is important in deep learning because it has been empirically shown that models operating on <strong>normalized</strong> data––whether it be inputs or latent activations––enjoy <em>faster convergence</em>.</p>
  <p>Unfortunately, zero-centered activation functions like <code>tanh</code> <strong>saturate</strong> at their <strong>asymptotes</strong> –– the gradients within this region get <strong>vanishingly smaller</strong> over time, leading to a weak training signal.</p>
  <p><code>ReLU</code> avoids this problem but it is not zero-centered. Therefore all-positive or all-negative activation functions whether <code>sigmoid</code> or <code>ReLU</code> can be difficult for <strong>gradient-based</strong> optimization. So, To solve this problem deep learning practitioners have invented a myriad of <em>Normalization</em> layers (<strong>batch norm, layer norm, weight norm</strong>, etc.). we can <strong>normalize the data in advance to be zero-centered</strong> as in batch/layer normalization.</p>
  <p><strong>Reference:</strong></p>
  <p><a href="https://arxiv.org/abs/2004.06632" rel="nofollow noreferrer"><em>A Survey on Activation Functions and their relation with Xavier and He Normal Initialization</em></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-28T15:37:06.683Z
Answers.Text: Thank you for your answer. Can you provide more details please ? I don't understand how if the $y=f(x^{T}w)$ function isn't *zero-centered* then the outputs will always be either positive or negative. If we can imagine that $f$ has a symetrical graph (like a gaussian function) shifted towards the positives for example, then it'll all depends on the "width" am I wrong ? A more concrete example: $y=tanh(z-2)$. Then if $x^{T}w=[-4,2,4]$ we'll have $tanh(x^{T}w)=[-1,0,1]$. Do we always expect that $x^{T}w$ will be very close to zero so that we aren't in the later case?
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-28T16:50:28.163Z
Answers.Text: like For the sigmoid function, the gradient saturates as $σ(x)→1$ or $σ(x)→0$ . We can simply tell this by the slope of the function as these points are approached, and also compute the derivative. We know that $σ′(x)=σ(x)(1−σ(x))$  , so $σ′(x)→0$ at these extreme values.
Answers.UserDisplayName: Faizy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-28T16:58:48.333Z
Answers.Text: Therefore, if inputs are `not zero-centered` and with sigmoid , saturation will be more likely than if our inputs were zero-centered. but early on in learning, we generally want **larger gradients**, since our error is high. This can drastically slow down $\eta$. Since `zero-centering` the data will bring the inputs of the sigmoid away from where they can **saturate**, `zero-centering` could increase the gradient signal during the early stages of learning, providing for faster learning.
Answers.UserDisplayName: Faizy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-28T17:21:46.577Z
Answers.Text: you know that for sigmoid the bound in the range of (0,1). Hence it always produces a non-negative value as output.
Answers.UserDisplayName: Faizy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-28T18:25:22.113Z
Answers.Text: This maybe somewhat out of context but let me ask it, "For negative values, Relu activation gives zero as output. Is it be a problem be for negative values weights of that layer will not update just like vanishing gradient of Tanh activation?
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-28T18:42:18.693Z
Answers.Text: @Faizy thank you for your answer. I think the term zero-centered may be a little confusing because we may have zero-centered data but that have big magnitude, imagine the set of points $\{ -10, -5, -3, 3, 5, 10\}$. Then we may say that this set is zero-centered, and it'll still give us saturation. But I understand what you mean. Thank you for all the explanation.
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-28T18:47:57.017Z
Answers.Text: @SwaksharDeb I don't think the problem comes from the weights alone since you have $tanh(x^{T}w)$. As long as $x^{T}w$ is negative that neuron won't be updated since the gradient will be zero. I believe this happens for two reasons: *1)* When you initialize the weights, you're unfortunate to have $x^{T}w$ negative and you get no update. *2)* You have good initialization but you update the weights in such a way that $x^{T}w$ become negative and you're neuron just dies. But since we're only able to work on our weights I guess you're right.
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-28T14:44:07.887Z
Answers.LastActivityDate: 2022-05-31T00:29:00.340Z
Answers.LastEditDate: 2022-05-31T00:29:00.340Z
Answers.UserDisplayName: Faizy
Body: <p>In <a href="https://youtu.be/wEoyxE0GP2M?list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&amp;t=524" rel="nofollow noreferrer">this lecture</a>, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that the gradient of our loss w.r.t. the weights <span class="math-container">$\frac{\partial L}{\partial w}$</span> which is equal to <span class="math-container">$\frac{\partial L}{\partial \sigma}\frac{\partial \sigma}{\partial w}$</span> will always be either negative or positive and we'll have a problem updating our weights as she shows in <a href="https://youtu.be/wEoyxE0GP2M?list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&amp;t=582" rel="nofollow noreferrer">this slide</a>, we won't be able to move in the direction of the vector <span class="math-container">$(1,-1)$</span>. I don't understand why since she only talks about one component of our gradient and not the whole vector. if the components of the gradient of our loss will have different signs which will allow us to adjust to different directions I'm I wrong? But the thing that I don't understand is how this property generalizes to non zero-centered functions and non-zero centered data?</p>
  
Tags: <backpropagation><activation-functions><sigmoid><gradient>
Title: Why is it a problem if the outputs of an activation function are not zero-centered?
UserDisplayName: Daviiid
ThreadText: # Why is it a problem if the outputs of an activation function are not zero-centered?
  
  (Asked by: username_0 on 2021-03-23)
  
  In [this lecture](https://youtu.be/wEoyxE0GP2M?list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&t=524), the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that the gradient of our loss w.r.t. the weights $\frac{\partial L}{\partial w}$ which is equal to $\frac{\partial L}{\partial \sigma}\frac{\partial \sigma}{\partial w}$ will always be either negative or positive and we'll have a problem updating our weights as she shows in [this slide](https://youtu.be/wEoyxE0GP2M?list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&t=582), we won't be able to move in the direction of the vector $(1,-1)$. I don't understand why since she only talks about one component of our gradient and not the whole vector. if the components of the gradient of our loss will have different signs which will allow us to adjust to different directions I'm I wrong? But the thing that I don't understand is how this property generalizes to non zero-centered functions and non-zero centered data?
  
  **Tags:** backpropagation, activation-functions, sigmoid, gradient
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-28. Score: 3 | Accepted answer
  
  Yes, if the activation function of the network is **not zero centered**, $y = f(x^{T}w)$ is always *positive* or always *negative*. Thus, the output of a layer is always being moved to either the positive values or the negative values. As a result, the **weight vector** needs **more updates** to be trained properly, and the number of **epochs** needed for the network to get trained also **increases**. This is why the **zero centered property is important**, though it is **NOT** necessary.
  
  **Zero-centered activation functions** ensure that the **mean activation value** is around **zero**. This property is important in deep learning because it has been empirically shown that models operating on **normalized** data––whether it be inputs or latent activations––enjoy *faster convergence*.
  
  Unfortunately, zero-centered activation functions like `tanh` **saturate** at their **asymptotes** –– the gradients within this region get **vanishingly smaller** over time, leading to a weak training signal.
  
  `ReLU` avoids this problem but it is not zero-centered. Therefore all-positive or all-negative activation functions whether `sigmoid` or `ReLU` can be difficult for **gradient-based** optimization. So, To solve this problem deep learning practitioners have invented a myriad of *Normalization* layers (**batch norm, layer norm, weight norm**, etc.). we can **normalize the data in advance to be zero-centered** as in batch/layer normalization.
  
  **Reference:**
  
  [*A Survey on Activation Functions and their relation with Xavier and He Normal Initialization*](https://arxiv.org/abs/2004.06632)
  
  > **Comments:**
  > **username_0**: Thank you for your answer. Can you provide more details please ? I don't understand how if the $y=f(x^{T}w)$ function isn't *zero-centered* then the outputs will always be either positive or negative. If we can imagine that $f$ has a symetrical graph (like a gaussian function) shifted towards the positives for example, then it'll all depends on the "width" am I wrong ? A more concrete example: $y=tanh(z-2)$. Then if $x^{T}w=[-4,2,4]$ we'll have $tanh(x^{T}w)=[-1,0,1]$. Do we always expect that $x^{T}w$ will be very close to zero so that we aren't in the later case?  
  > **username_1**: like For the sigmoid function, the gradient saturates as $σ(x)→1$ or $σ(x)→0$ . We can simply tell this by the slope of the function as these points are approached, and also compute the derivative. We know that $σ′(x)=σ(x)(1−σ(x))$  , so $σ′(x)→0$ at these extreme values.  
  > **username_1**: Therefore, if inputs are `not zero-centered` and with sigmoid , saturation will be more likely than if our inputs were zero-centered. but early on in learning, we generally want **larger gradients**, since our error is high. This can drastically slow down $\eta$. Since `zero-centering` the data will bring the inputs of the sigmoid away from where they can **saturate**, `zero-centering` could increase the gradient signal during the early stages of learning, providing for faster learning.  
  > **username_1**: you know that for sigmoid the bound in the range of (0,1). Hence it always produces a non-negative value as output.  
  > **username_2**: This maybe somewhat out of context but let me ask it, "For negative values, Relu activation gives zero as output. Is it be a problem be for negative values weights of that layer will not update just like vanishing gradient of Tanh activation?  
  > **username_0**: @username_1 thank you for your answer. I think the term zero-centered may be a little confusing because we may have zero-centered data but that have big magnitude, imagine the set of points $\{ -10, -5, -3, 3, 5, 10\}$. Then we may say that this set is zero-centered, and it'll still give us saturation. But I understand what you mean. Thank you for all the explanation.  
  > **username_0**: @username_2 I don't think the problem comes from the weights alone since you have $tanh(x^{T}w)$. As long as $x^{T}w$ is negative that neuron won't be updated since the gradient will be zero. I believe this happens for two reasons: *1)* When you initialize the weights, you're unfortunate to have $x^{T}w$ negative and you get no update. *2)* You have good initialization but you update the weights in such a way that $x^{T}w$ become negative and you're neuron just dies. But since we're only able to work on our weights I guess you're right.  
  
  ---
  
  