<p>This is the case as the loss doesn't have to monotonically decrease when it's updated in the negative direction.
For example:</p>
<p>Let <span class="math-container">$L(\theta) = \theta^2 $</span>  and <span class="math-container">$\theta_0= 3$</span></p>
<p>Let the subscript n in <span class="math-container">$\theta_n$</span> denote the iteration number.</p>
<p>Then <span class="math-container">$\nabla_{\theta}L(\theta_0) = 2*\theta = 2*3 = 6$</span></p>
<p>For the loss to decrease in this case <span class="math-container">$\epsilon &lt; 1$</span> needs to hold so lets look at when it doesn't hold.</p>
<p>If <span class="math-container">$\epsilon = 1$</span> then the loss would be stable at 6 as the update would give <span class="math-container">$\theta_1 = \theta_0 - 1*\nabla_{\theta}L(\theta_0) = 3-6 = -3 $</span></p>
<p><span class="math-container">$\theta_2 = \theta_1 - 1*\nabla_{\theta}L(\theta_1)  = -3 -- 6 = 3$</span></p>
<p><span class="math-container">$\theta_3 = \theta_2 - 1*\nabla_{\theta}L(\theta_2) = 3 - 6 = -3$</span></p>
<p><span class="math-container">$L(\theta_1) = L(\theta_2) =L(\theta_3) = 6$</span>
As <span class="math-container">$L(-3) = (-3)^2 = 6$</span> and <span class="math-container">$L(3) = 3^2 = 6$</span></p>
<p>So if <span class="math-container">$\epsilon = 1$</span> then <span class="math-container">$\theta$</span> will oscillate back and forth between 3 and -3 what gives a loss that is stable at 6.</p>
<p>Now if we look at what happens if <span class="math-container">$\epsilon &gt;1$</span> for example <span class="math-container">$\epsilon =2$</span></p>
<p>note that <span class="math-container">$\nabla_{\theta}L(\theta) = 2*\theta$</span> as <span class="math-container">$L(\theta) = \theta^2$</span></p>
<p><span class="math-container">$\theta_1 = \theta_0 - 2*\nabla_{\theta}L(\theta_0) =  3 -2*\nabla_{\theta}L(\theta_0)  = 3 - 2 * (2*3) = 3 - 12 = -9$</span></p>
<p><span class="math-container">$\theta_2 = \theta_1 - 2*\nabla_{\theta}L(\theta_1)  = -9 - 2 *\nabla_{\theta}L(\theta_1) = -9-2*(2*-9) = -9 + 36 = 27$</span></p>
<p><span class="math-container">$\theta_3 = \theta_2 - 2*\nabla_{\theta}L(\theta_2)  = 27 - 2 *\nabla_{\theta}L(\theta_2) = 27 -2 * (2*27) = 27 -108 = -81 $</span></p>
<p>In  turn the loss would  increase after each update .</p>
<p><span class="math-container">$L(\theta_1) = (-9)^2 = 81$</span></p>
<p><span class="math-container">$L(\theta_2) = 27^2 = 729$</span></p>
<p><span class="math-container">$(\theta_3) = (-81)^2 = 6561$</span></p>

CC BY-SA 4.0
2022-01-10T10:57:54.513Z
The explanations in this answer seem correct to me. However, the learning rate is usually a number in the range $(0, 1)$ (see [this answer](https://ai.stackexchange.com/a/23748/2444)), so maybe, in that excerpt, the authors were already assuming that the learning rate is not bigger or equal to 1. So, you may want to comment on that case too.
nbro
CC BY-SA 4.0
2022-01-10T11:06:05.330Z
The cited text says when it's too large, so commonly used learning rates don't apply, but I get what you mean. The actual chosen value of epsilon fully depends on the behavior of the loss landscape.
hal9000
CC BY-SA 4.0
2022-01-10T10:39:48.380Z
2022-01-10T10:39:48.380Z
hal9000
<p>In Deep Learning by Goodfellow et al., I came across the following line on the chapter on Stochastic Gradient Descent (pg. 287):</p>
<blockquote>
<p>The main question is how to set <span class="math-container">$\epsilon_0$</span>. If it is too large, the
learning curve will show violent oscillations, with the cost function
often increasing significantly.</p>
</blockquote>
<p>I'm slightly confused why the loss function would increase at all. My understanding of gradient descent is that given parameters <span class="math-container">$\theta$</span> and a loss function <span class="math-container">$\ell (\vec{\theta})$</span>, the gradient update is performed as follows:</p>
<p><span class="math-container">$$\vec{\theta}_{t+1} = \vec{\theta}_{t} - \epsilon \nabla_{\vec{\theta}}\ell (\vec{\theta})$$</span></p>
<p>The loss function is guaranteed to monotonically decrease because the parameters are updated in the negative direction of the gradient. I would assume the same holds for SGD, but clearly it doesn't. With a high learning rate <span class="math-container">$\epsilon$</span>, how would the loss function increase in its value? Is my interpretation incorrect, or does SGD have different theoretical guarantees than vanilla gradient descent?</p>

<neural-networks><machine-learning><deep-learning><gradient-descent><learning-rate>
Why can the learning rate make the loss increase in stochastic gradient descent?
Shrey
# Why can the learning rate make the loss increase in stochastic gradient descent?

(Asked by: username_0 on 2019-01-10)

In Deep Learning by Goodfellow et al., I came across the following line on the chapter on Stochastic Gradient Descent (pg. 287):

> The main question is how to set $\epsilon\_0$. If it is too large, the
> learning curve will show violent oscillations, with the cost function
> often increasing significantly.

I'm slightly confused why the loss function would increase at all. My understanding of gradient descent is that given parameters $\theta$ and a loss function $\ell (\vec{\theta})$, the gradient update is performed as follows:

$$\vec{\theta}\_{t+1} = \vec{\theta}\_{t} - \epsilon \nabla\_{\vec{\theta}}\ell (\vec{\theta})$$

The loss function is guaranteed to monotonically decrease because the parameters are updated in the negative direction of the gradient. I would assume the same holds for SGD, but clearly it doesn't. With a high learning rate $\epsilon$, how would the loss function increase in its value? Is my interpretation incorrect, or does SGD have different theoretical guarantees than vanilla gradient descent?

**Tags:** neural-networks, machine-learning, deep-learning, gradient-descent, learning-rate


---

## 1 answers

### Answer by username_1 on 2022-01-10. Score: 1

This is the case as the loss doesn't have to monotonically decrease when it's updated in the negative direction.
For example:

Let $L(\theta) = \theta^2 $ and $\theta\_0= 3$

Let the subscript n in $\theta\_n$ denote the iteration number.

Then $\nabla\_{\theta}L(\theta\_0) = 2\*\theta = 2\*3 = 6$

For the loss to decrease in this case $\epsilon < 1$ needs to hold so lets look at when it doesn't hold.

If $\epsilon = 1$ then the loss would be stable at 6 as the update would give $\theta\_1 = \theta\_0 - 1\*\nabla\_{\theta}L(\theta\_0) = 3-6 = -3 $

$\theta\_2 = \theta\_1 - 1\*\nabla\_{\theta}L(\theta\_1) = -3 -- 6 = 3$

$\theta\_3 = \theta\_2 - 1\*\nabla\_{\theta}L(\theta\_2) = 3 - 6 = -3$

$L(\theta\_1) = L(\theta\_2) =L(\theta\_3) = 6$
As $L(-3) = (-3)^2 = 6$ and $L(3) = 3^2 = 6$

So if $\epsilon = 1$ then $\theta$ will oscillate back and forth between 3 and -3 what gives a loss that is stable at 6.

Now if we look at what happens if $\epsilon >1$ for example $\epsilon =2$

note that $\nabla\_{\theta}L(\theta) = 2\*\theta$ as $L(\theta) = \theta^2$

$\theta\_1 = \theta\_0 - 2\*\nabla\_{\theta}L(\theta\_0) = 3 -2\*\nabla\_{\theta}L(\theta\_0) = 3 - 2 \* (2\*3) = 3 - 12 = -9$

$\theta\_2 = \theta\_1 - 2\*\nabla\_{\theta}L(\theta\_1) = -9 - 2 \*\nabla\_{\theta}L(\theta\_1) = -9-2\*(2\*-9) = -9 + 36 = 27$

$\theta\_3 = \theta\_2 - 2\*\nabla\_{\theta}L(\theta\_2) = 27 - 2 \*\nabla\_{\theta}L(\theta\_2) = 27 -2 \* (2\*27) = 27 -108 = -81 $

In turn the loss would increase after each update .

$L(\theta\_1) = (-9)^2 = 81$

$L(\theta\_2) = 27^2 = 729$

$(\theta\_3) = (-81)^2 = 6561$

> **Comments:**
> **username_2**: The explanations in this answer seem correct to me. However, the learning rate is usually a number in the range $(0, 1)$ (see [this answer](https://ai.stackexchange.com/a/23748/2444)), so maybe, in that excerpt, the authors were already assuming that the learning rate is not bigger or equal to 1. So, you may want to comment on that case too.  
> **username_1**: The cited text says when it's too large, so commonly used learning rates don't apply, but I get what you mean. The actual chosen value of epsilon fully depends on the behavior of the loss landscape.  

---

