Answers.Body: <p>Let me try to explain here.</p>
  <p>Usually, we calculate the variance by subtracting the mean term and then square it. But here mean (first-moment m_t) is fluctuating like anything at each time &quot;t&quot; and is getting calculated with the influence of past mean as well, also with the influence of beta_1.</p>
  <p>So when the 2nd-moment term v_t is getting calculated with a similar approach, when variance is mentioned in the Adam context is understood that we are considering the gradient mean to be equal to zero. This makes sense, since there is no reason to think positive or negative gradients would be more probable than one-another. we are only considering g_t^2 here, so this is called Uncentred variance in the context of calculating 2nd momentum for ADAM.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-11T13:46:10.847Z
Answers.LastActivityDate: 2022-01-19T15:13:25.753Z
Answers.LastEditDate: 2022-01-19T15:13:25.753Z
Answers.UserDisplayName: Kishan Kumar
Body: <p>I have been reading about Adam and AdamW (<a href="https://towardsdatascience.com/why-adamw-matters-736223f31b5d" rel="nofollow noreferrer">Here</a>). The author mentioned that in &quot;uncentered variance&quot; we don't consider subtracting mean</p>
  <p><a href="https://i.sstatic.net/WSn4p.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WSn4p.png" alt="enter image description here" /></a></p>
  <p>In this statement, the author is talking about uncentered variance and how it becomes equal to the square of the mean.</p>
  <p><a href="https://i.sstatic.net/lQJUa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lQJUa.png" alt="enter image description here" /></a></p>
  <p><strong>I want to understand what exactly is uncentered variance</strong>? (because if I consider the general equation of variance <span class="math-container">$\dfrac{(\text{obs-mean})^2}{(n-1)}$</span> then it is not making sense to me the removing mean will lead to the definition of uncentered variance as we need a point around which we are calculating variance and here mean is that point)</p>
  <p><strong>Also if we are making mean=0(not subtracting mean from obs) then if I consider this as
  uncentered mean (For me it is variance around 0) then it is becoming hard to understand how this will lead to uncentered <span class="math-container">$\text{variance = mean}^2$</span></strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-22T13:45:24.977Z
Comments.Text: Hello. Rather than providing a screenshot of the excerpt that you're quoting, it would be better if you can copy and paste the text directly into your post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-22T17:10:49.960Z
Comments.Text: Thanks for the suggestion. I'll follow it next time. Just curious to know what difference text makes in comparison to screenshot?
Comments.UserDisplayName: learner
Tags: <neural-networks><deep-learning><optimization><gradient-descent><optimizers>
Title: What is uncentered variance and how it becomes equal to mean square in Adam?
UserDisplayName: learner
ThreadText: # What is uncentered variance and how it becomes equal to mean square in Adam?
  
  (Asked by: username_0 on 2021-10-20)
  
  I have been reading about Adam and AdamW ([Here](https://towardsdatascience.com/why-adamw-matters-736223f31b5d)). The author mentioned that in "uncentered variance" we don't consider subtracting mean
  
  [![enter image description here](https://i.sstatic.net/WSn4p.png)](https://i.sstatic.net/WSn4p.png)
  
  In this statement, the author is talking about uncentered variance and how it becomes equal to the square of the mean.
  
  [![enter image description here](https://i.sstatic.net/lQJUa.png)](https://i.sstatic.net/lQJUa.png)
  
  **I want to understand what exactly is uncentered variance**? (because if I consider the general equation of variance $\dfrac{(\text{obs-mean})^2}{(n-1)}$ then it is not making sense to me the removing mean will lead to the definition of uncentered variance as we need a point around which we are calculating variance and here mean is that point)
  
  **Also if we are making mean=0(not subtracting mean from obs) then if I consider this as
  uncentered mean (For me it is variance around 0) then it is becoming hard to understand how this will lead to uncentered $\text{variance = mean}^2$**
  
  **Tags:** neural-networks, deep-learning, optimization, gradient-descent, optimizers
  > **Comments:**
  > **username_1**: Hello. Rather than providing a screenshot of the excerpt that you're quoting, it would be better if you can copy and paste the text directly into your post.  
  > **username_0**: Thanks for the suggestion. I'll follow it next time. Just curious to know what difference text makes in comparison to screenshot?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-01-11. Score: 3
  
  Let me try to explain here.
  
  Usually, we calculate the variance by subtracting the mean term and then square it. But here mean (first-moment m\_t) is fluctuating like anything at each time "t" and is getting calculated with the influence of past mean as well, also with the influence of beta\_1.
  
  So when the 2nd-moment term v\_t is getting calculated with a similar approach, when variance is mentioned in the Adam context is understood that we are considering the gradient mean to be equal to zero. This makes sense, since there is no reason to think positive or negative gradients would be more probable than one-another. we are only considering g\_t^2 here, so this is called Uncentred variance in the context of calculating 2nd momentum for ADAM.
  
  
  ---
  
  