Answers.Body: <p>The authors describe their belief in Section 3:</p>
  <blockquote>
  <p>Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima.</p>
  </blockquote>
  <p>Diving further, in section 3.2:</p>
  <blockquote>
  <p>Adam uses the exponential moving average to calculate the adaptive learning rate.  For gradients <span class="math-container">$\{g_1,\dots,g_t\}$</span>, their exponential moving average has a larger variance than their simple average.  Also, in the early stage (<span class="math-container">$t$</span> is small), the difference of the exponential weights of <span class="math-container">$\{g_1,\dots,g_t\}$</span> is relatively small (up to <span class="math-container">$1−β^{t−1}_2$</span>).</p>
  </blockquote>
  <p>It seems like the root issue is that exponential moving average, while great with many data samples, have too large of a variance with few data samples. It is this variance that sometimes allow the gradient descend in a bad optima.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-11T06:56:16.963Z
Answers.LastActivityDate: 2020-12-11T06:56:16.963Z
Answers.UserDisplayName: user3667125
Body: <p>In the paper <a href="https://arxiv.org/abs/1908.03265" rel="nofollow noreferrer">On the Variance of the Adaptive Learning Rate and Beyond</a>, in section 2, the authors write</p>
  <blockquote>
  <p>To further analyze this phenomenon, we visualize the histogram of the absolute value of gradients on a log scale in Figure 2. We observe that, without applying warmup, the gradient distribution is distorted to have a mass center in relatively small values within 10 updates. Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.</p>
  </blockquote>
  <p>Here is figure 2 from the paper.</p>
  <p><a href="https://i.sstatic.net/JbhuE.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JbhuE.png" alt="enter image description here" /></a></p>
  <p>Can someone explain this part?</p>
  <blockquote>
  <p>Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.</p>
  </blockquote>
  <p>Why is this true?</p>
  
Tags: <deep-learning><papers><optimization><learning-rate><adam>
Title: Why is Adam trapped in bad/suspicious local optima after the first few updates?
UserDisplayName: AmorFati
ThreadText: # Why is Adam trapped in bad/suspicious local optima after the first few updates?
  
  (Asked by: username_0 on 2020-12-10)
  
  In the paper [On the Variance of the Adaptive Learning Rate and Beyond](https://arxiv.org/abs/1908.03265), in section 2, the authors write
  
  > To further analyze this phenomenon, we visualize the histogram of the absolute value of gradients on a log scale in Figure 2. We observe that, without applying warmup, the gradient distribution is distorted to have a mass center in relatively small values within 10 updates. Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.
  
  Here is figure 2 from the paper.
  
  [![enter image description here](https://i.sstatic.net/JbhuE.png)](https://i.sstatic.net/JbhuE.png)
  
  Can someone explain this part?
  
  > Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.
  
  Why is this true?
  
  **Tags:** deep-learning, papers, optimization, learning-rate, adam
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-11. Score: 1
  
  The authors describe their belief in Section 3:
  
  > Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima.
  
  Diving further, in section 3.2:
  
  > Adam uses the exponential moving average to calculate the adaptive learning rate. For gradients $\{g\_1,\dots,g\_t\}$, their exponential moving average has a larger variance than their simple average. Also, in the early stage ($t$ is small), the difference of the exponential weights of $\{g\_1,\dots,g\_t\}$ is relatively small (up to $1−β^{t−1}\_2$).
  
  It seems like the root issue is that exponential moving average, while great with many data samples, have too large of a variance with few data samples. It is this variance that sometimes allow the gradient descend in a bad optima.
  
  
  ---
  
  