Answers.Body: <p>This is the case as the loss doesn't have to monotonically decrease when it's updated in the negative direction.
  For example:</p>
  <p>Let <span class="math-container">$L(\theta) = \theta^2 $</span>  and <span class="math-container">$\theta_0= 3$</span></p>
  <p>Let the subscript n in <span class="math-container">$\theta_n$</span> denote the iteration number.</p>
  <p>Then <span class="math-container">$\nabla_{\theta}L(\theta_0) = 2*\theta = 2*3 = 6$</span></p>
  <p>For the loss to decrease in this case <span class="math-container">$\epsilon &lt; 1$</span> needs to hold so lets look at when it doesn't hold.</p>
  <p>If <span class="math-container">$\epsilon = 1$</span> then the loss would be stable at 6 as the update would give <span class="math-container">$\theta_1 = \theta_0 - 1*\nabla_{\theta}L(\theta_0) = 3-6 = -3 $</span></p>
  <p><span class="math-container">$\theta_2 = \theta_1 - 1*\nabla_{\theta}L(\theta_1)  = -3 -- 6 = 3$</span></p>
  <p><span class="math-container">$\theta_3 = \theta_2 - 1*\nabla_{\theta}L(\theta_2) = 3 - 6 = -3$</span></p>
  <p><span class="math-container">$L(\theta_1) = L(\theta_2) =L(\theta_3) = 6$</span>
  As <span class="math-container">$L(-3) = (-3)^2 = 6$</span> and <span class="math-container">$L(3) = 3^2 = 6$</span></p>
  <p>So if <span class="math-container">$\epsilon = 1$</span> then <span class="math-container">$\theta$</span> will oscillate back and forth between 3 and -3 what gives a loss that is stable at 6.</p>
  <p>Now if we look at what happens if <span class="math-container">$\epsilon &gt;1$</span> for example <span class="math-container">$\epsilon =2$</span></p>
  <p>note that <span class="math-container">$\nabla_{\theta}L(\theta) = 2*\theta$</span> as <span class="math-container">$L(\theta) = \theta^2$</span></p>
  <p><span class="math-container">$\theta_1 = \theta_0 - 2*\nabla_{\theta}L(\theta_0) =  3 -2*\nabla_{\theta}L(\theta_0)  = 3 - 2 * (2*3) = 3 - 12 = -9$</span></p>
  <p><span class="math-container">$\theta_2 = \theta_1 - 2*\nabla_{\theta}L(\theta_1)  = -9 - 2 *\nabla_{\theta}L(\theta_1) = -9-2*(2*-9) = -9 + 36 = 27$</span></p>
  <p><span class="math-container">$\theta_3 = \theta_2 - 2*\nabla_{\theta}L(\theta_2)  = 27 - 2 *\nabla_{\theta}L(\theta_2) = 27 -2 * (2*27) = 27 -108 = -81 $</span></p>
  <p>In  turn the loss would  increase after each update .</p>
  <p><span class="math-container">$L(\theta_1) = (-9)^2 = 81$</span></p>
  <p><span class="math-container">$L(\theta_2) = 27^2 = 729$</span></p>
  <p><span class="math-container">$(\theta_3) = (-81)^2 = 6561$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-10T10:57:54.513Z
Answers.Text: The explanations in this answer seem correct to me. However, the learning rate is usually a number in the range $(0, 1)$ (see [this answer](https://ai.stackexchange.com/a/23748/2444)), so maybe, in that excerpt, the authors were already assuming that the learning rate is not bigger or equal to 1. So, you may want to comment on that case too.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-10T11:06:05.330Z
Answers.Text: The cited text says when it's too large, so commonly used learning rates don't apply, but I get what you mean. The actual chosen value of epsilon fully depends on the behavior of the loss landscape.
Answers.UserDisplayName: hal9000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-10T10:39:48.380Z
Answers.LastActivityDate: 2022-01-10T10:39:48.380Z
Answers.UserDisplayName: hal9000
Body: <p>In Deep Learning by Goodfellow et al., I came across the following line on the chapter on Stochastic Gradient Descent (pg. 287):</p>
  <blockquote>
  <p>The main question is how to set <span class="math-container">$\epsilon_0$</span>. If it is too large, the
  learning curve will show violent oscillations, with the cost function
  often increasing significantly.</p>
  </blockquote>
  <p>I'm slightly confused why the loss function would increase at all. My understanding of gradient descent is that given parameters <span class="math-container">$\theta$</span> and a loss function <span class="math-container">$\ell (\vec{\theta})$</span>, the gradient update is performed as follows:</p>
  <p><span class="math-container">$$\vec{\theta}_{t+1} = \vec{\theta}_{t} - \epsilon \nabla_{\vec{\theta}}\ell (\vec{\theta})$$</span></p>
  <p>The loss function is guaranteed to monotonically decrease because the parameters are updated in the negative direction of the gradient. I would assume the same holds for SGD, but clearly it doesn't. With a high learning rate <span class="math-container">$\epsilon$</span>, how would the loss function increase in its value? Is my interpretation incorrect, or does SGD have different theoretical guarantees than vanilla gradient descent?</p>
  
Tags: <neural-networks><machine-learning><deep-learning><gradient-descent><learning-rate>
Title: Why can the learning rate make the loss increase in stochastic gradient descent?
UserDisplayName: Shrey
ThreadText: # Why can the learning rate make the loss increase in stochastic gradient descent?
  
  (Asked by: username_0 on 2019-01-10)
  
  In Deep Learning by Goodfellow et al., I came across the following line on the chapter on Stochastic Gradient Descent (pg. 287):
  
  > The main question is how to set $\epsilon\_0$. If it is too large, the
  > learning curve will show violent oscillations, with the cost function
  > often increasing significantly.
  
  I'm slightly confused why the loss function would increase at all. My understanding of gradient descent is that given parameters $\theta$ and a loss function $\ell (\vec{\theta})$, the gradient update is performed as follows:
  
  $$\vec{\theta}\_{t+1} = \vec{\theta}\_{t} - \epsilon \nabla\_{\vec{\theta}}\ell (\vec{\theta})$$
  
  The loss function is guaranteed to monotonically decrease because the parameters are updated in the negative direction of the gradient. I would assume the same holds for SGD, but clearly it doesn't. With a high learning rate $\epsilon$, how would the loss function increase in its value? Is my interpretation incorrect, or does SGD have different theoretical guarantees than vanilla gradient descent?
  
  **Tags:** neural-networks, machine-learning, deep-learning, gradient-descent, learning-rate
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-10. Score: 1
  
  This is the case as the loss doesn't have to monotonically decrease when it's updated in the negative direction.
  For example:
  
  Let $L(\theta) = \theta^2 $ and $\theta\_0= 3$
  
  Let the subscript n in $\theta\_n$ denote the iteration number.
  
  Then $\nabla\_{\theta}L(\theta\_0) = 2\*\theta = 2\*3 = 6$
  
  For the loss to decrease in this case $\epsilon < 1$ needs to hold so lets look at when it doesn't hold.
  
  If $\epsilon = 1$ then the loss would be stable at 6 as the update would give $\theta\_1 = \theta\_0 - 1\*\nabla\_{\theta}L(\theta\_0) = 3-6 = -3 $
  
  $\theta\_2 = \theta\_1 - 1\*\nabla\_{\theta}L(\theta\_1) = -3 -- 6 = 3$
  
  $\theta\_3 = \theta\_2 - 1\*\nabla\_{\theta}L(\theta\_2) = 3 - 6 = -3$
  
  $L(\theta\_1) = L(\theta\_2) =L(\theta\_3) = 6$
  As $L(-3) = (-3)^2 = 6$ and $L(3) = 3^2 = 6$
  
  So if $\epsilon = 1$ then $\theta$ will oscillate back and forth between 3 and -3 what gives a loss that is stable at 6.
  
  Now if we look at what happens if $\epsilon >1$ for example $\epsilon =2$
  
  note that $\nabla\_{\theta}L(\theta) = 2\*\theta$ as $L(\theta) = \theta^2$
  
  $\theta\_1 = \theta\_0 - 2\*\nabla\_{\theta}L(\theta\_0) = 3 -2\*\nabla\_{\theta}L(\theta\_0) = 3 - 2 \* (2\*3) = 3 - 12 = -9$
  
  $\theta\_2 = \theta\_1 - 2\*\nabla\_{\theta}L(\theta\_1) = -9 - 2 \*\nabla\_{\theta}L(\theta\_1) = -9-2\*(2\*-9) = -9 + 36 = 27$
  
  $\theta\_3 = \theta\_2 - 2\*\nabla\_{\theta}L(\theta\_2) = 27 - 2 \*\nabla\_{\theta}L(\theta\_2) = 27 -2 \* (2\*27) = 27 -108 = -81 $
  
  In turn the loss would increase after each update .
  
  $L(\theta\_1) = (-9)^2 = 81$
  
  $L(\theta\_2) = 27^2 = 729$
  
  $(\theta\_3) = (-81)^2 = 6561$
  
  > **Comments:**
  > **username_2**: The explanations in this answer seem correct to me. However, the learning rate is usually a number in the range $(0, 1)$ (see [this answer](https://ai.stackexchange.com/a/23748/2444)), so maybe, in that excerpt, the authors were already assuming that the learning rate is not bigger or equal to 1. So, you may want to comment on that case too.  
  > **username_1**: The cited text says when it's too large, so commonly used learning rates don't apply, but I get what you mean. The actual chosen value of epsilon fully depends on the behavior of the loss landscape.  
  
  ---
  
  