<p>The authors describe their belief in Section 3:</p>
<blockquote>
<p>Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima.</p>
</blockquote>
<p>Diving further, in section 3.2:</p>
<blockquote>
<p>Adam uses the exponential moving average to calculate the adaptive learning rate.  For gradients <span class="math-container">$\{g_1,\dots,g_t\}$</span>, their exponential moving average has a larger variance than their simple average.  Also, in the early stage (<span class="math-container">$t$</span> is small), the difference of the exponential weights of <span class="math-container">$\{g_1,\dots,g_t\}$</span> is relatively small (up to <span class="math-container">$1−β^{t−1}_2$</span>).</p>
</blockquote>
<p>It seems like the root issue is that exponential moving average, while great with many data samples, have too large of a variance with few data samples. It is this variance that sometimes allow the gradient descend in a bad optima.</p>

CC BY-SA 4.0
2020-12-11T06:56:16.963Z
2020-12-11T06:56:16.963Z
user3667125
<p>In the paper <a href="https://arxiv.org/abs/1908.03265" rel="nofollow noreferrer">On the Variance of the Adaptive Learning Rate and Beyond</a>, in section 2, the authors write</p>
<blockquote>
<p>To further analyze this phenomenon, we visualize the histogram of the absolute value of gradients on a log scale in Figure 2. We observe that, without applying warmup, the gradient distribution is distorted to have a mass center in relatively small values within 10 updates. Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.</p>
</blockquote>
<p>Here is figure 2 from the paper.</p>
<p><a href="https://i.sstatic.net/JbhuE.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JbhuE.png" alt="enter image description here" /></a></p>
<p>Can someone explain this part?</p>
<blockquote>
<p>Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.</p>
</blockquote>
<p>Why is this true?</p>

<deep-learning><papers><optimization><learning-rate><adam>
Why is Adam trapped in bad/suspicious local optima after the first few updates?
AmorFati
# Why is Adam trapped in bad/suspicious local optima after the first few updates?

(Asked by: username_0 on 2020-12-10)

In the paper [On the Variance of the Adaptive Learning Rate and Beyond](https://arxiv.org/abs/1908.03265), in section 2, the authors write

> To further analyze this phenomenon, we visualize the histogram of the absolute value of gradients on a log scale in Figure 2. We observe that, without applying warmup, the gradient distribution is distorted to have a mass center in relatively small values within 10 updates. Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.

Here is figure 2 from the paper.

[![enter image description here](https://i.sstatic.net/JbhuE.png)](https://i.sstatic.net/JbhuE.png)

Can someone explain this part?

> Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.

Why is this true?

**Tags:** deep-learning, papers, optimization, learning-rate, adam


---

## 1 answers

### Answer by username_1 on 2020-12-11. Score: 1

The authors describe their belief in Section 3:

> Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima.

Diving further, in section 3.2:

> Adam uses the exponential moving average to calculate the adaptive learning rate. For gradients $\{g\_1,\dots,g\_t\}$, their exponential moving average has a larger variance than their simple average. Also, in the early stage ($t$ is small), the difference of the exponential weights of $\{g\_1,\dots,g\_t\}$ is relatively small (up to $1−β^{t−1}\_2$).

It seems like the root issue is that exponential moving average, while great with many data samples, have too large of a variance with few data samples. It is this variance that sometimes allow the gradient descend in a bad optima.


---

