Answers.Body: <p>Actually it won't go to 7, but will go to somewhere less far away. Because the  previous gradients at 5 and before are less steep, the accumulated gradient with momentum will be smaller and it will not shoot as far. The end effect is less overshoot and so it helps in this case.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-10T18:40:16.587Z
Answers.LastActivityDate: 2023-02-10T18:40:16.587Z
Answers.UserDisplayName: user559678
Answers.Body: <p>Your concern is valid that's why there's another <a href="https://paperswithcode.com/method/nesterov-accelerated-gradient" rel="nofollow noreferrer">Nesterov accelerated gradient method</a> specifically proposed to further address such edge case where various momentum based methods using past and current gradients may still overshoot at point 6 and even beyond. Nesterov momentum is essentially a form of one-step look ahead momentum which can reduce the amount of oscillation near your steep global/local minima.</p>
  <p>However, with the usually very small learning rate the vanilla momentum methods have much better convergence rate near minima in general compared to traditional gradient descent first order methods. And as your source correctly claims in your above sharp minimum rare cases, momentum based optimization methods will obviously dampen the updated amount at each iteration point, thus <em>comparing</em> to traditional gradient descent methods they'll tend to not overshoot at point 6 and beyond.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-06T22:15:46.587Z
Answers.LastActivityDate: 2024-07-06T22:15:46.587Z
Answers.UserDisplayName: cinch
Answers.Body: <p>What your professor most probably means is that since momentum adds your previous gradient to the current (as in a moving average). The speed built up till point 7 (moving in to the right) will added to the gradient from 7 to 8 that is going to the left, causing them to cancel each other out (what might cause the update to turn small enough for the loss to converge instead of explode).
  So it helps at the point where the gradient suddenly changes direction, as the momentum will cancel it out partially.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T14:42:17.943Z
Answers.LastActivityDate: 2022-06-21T14:42:17.943Z
Answers.UserDisplayName: hal9000
Body: <p>Classical gradient descent algorithms sometimes overshoot and escape minima as they depend on the gradient only. You can see such a problem during the update from point 6.</p>
  <p><a href="https://i.sstatic.net/XJu8M.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XJu8M.jpg" alt="enter image description here" /></a></p>
  <p>In classical GD algorithm, the update equation is</p>
  <p><span class="math-container">$$\theta_{t+1} = \theta_{t} - \eta \times \triangledown_{\theta} \ell$$</span></p>
  <p>In the momentum based GD algorithm, the update equations are</p>
  <p><span class="math-container">$$v_0 = 0$$</span>
  <span class="math-container">$$v_{t+1} = \alpha v_t +  \eta \times \triangledown_{\theta} \ell $$</span>
  <span class="math-container">$$\theta = \theta - \eta \times \triangledown_{\theta} \ell$$</span></p>
  <p>I am writing all the equations concisely by removing the obvious variables used such as inputs to loss functions. In the lecture I'm listening to, the narrator says that <strong>momentum-based GD helps during the update at point 6 and the update will not lead to point 7 as shown in the figure and goes towards minima.</strong></p>
  <p>But for me, it seems that even momentum-based GD will go to point 7 and the update at point 7 will be benefited from the momentum-based GD as it does not lead to point 8 and goes towards minima.</p>
  <p>Am I correct? If not, at which point does the momentum-based GD actually help?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-07T08:23:26.090Z
Comments.Text: Your momentum update equations look wrong. Importantly, the last one which updates parameters is not modified from basic gradient descent. It doesn't use the momentum. Could you check whether this is really the version you want to ask about?
Comments.UserDisplayName: Neil Slater
Tags: <gradient-descent><momentum>
Title: At which point, does the momentum based GD helps really in this figure?
UserDisplayName: hanugm
ThreadText: # At which point, does the momentum based GD helps really in this figure?
  
  (Asked by: username_0 on 2022-05-24)
  
  Classical gradient descent algorithms sometimes overshoot and escape minima as they depend on the gradient only. You can see such a problem during the update from point 6.
  
  [![enter image description here](https://i.sstatic.net/XJu8M.jpg)](https://i.sstatic.net/XJu8M.jpg)
  
  In classical GD algorithm, the update equation is
  
  $$\theta\_{t+1} = \theta\_{t} - \eta \times \triangledown\_{\theta} \ell$$
  
  In the momentum based GD algorithm, the update equations are
  
  $$v\_0 = 0$$
  $$v\_{t+1} = \alpha v\_t + \eta \times \triangledown\_{\theta} \ell $$
  $$\theta = \theta - \eta \times \triangledown\_{\theta} \ell$$
  
  I am writing all the equations concisely by removing the obvious variables used such as inputs to loss functions. In the lecture I'm listening to, the narrator says that **momentum-based GD helps during the update at point 6 and the update will not lead to point 7 as shown in the figure and goes towards minima.**
  
  But for me, it seems that even momentum-based GD will go to point 7 and the update at point 7 will be benefited from the momentum-based GD as it does not lead to point 8 and goes towards minima.
  
  Am I correct? If not, at which point does the momentum-based GD actually help?
  
  **Tags:** gradient-descent, momentum
  > **Comments:**
  > **username_1**: Your momentum update equations look wrong. Importantly, the last one which updates parameters is not modified from basic gradient descent. It doesn't use the momentum. Could you check whether this is really the version you want to ask about?  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2023-02-10. Score: 0
  
  Actually it won't go to 7, but will go to somewhere less far away. Because the previous gradients at 5 and before are less steep, the accumulated gradient with momentum will be smaller and it will not shoot as far. The end effect is less overshoot and so it helps in this case.
  
  
  ---
  
  ### Answer by username_3 on 2024-07-06. Score: 0
  
  Your concern is valid that's why there's another [Nesterov accelerated gradient method](https://paperswithcode.com/method/nesterov-accelerated-gradient) specifically proposed to further address such edge case where various momentum based methods using past and current gradients may still overshoot at point 6 and even beyond. Nesterov momentum is essentially a form of one-step look ahead momentum which can reduce the amount of oscillation near your steep global/local minima.
  
  However, with the usually very small learning rate the vanilla momentum methods have much better convergence rate near minima in general compared to traditional gradient descent first order methods. And as your source correctly claims in your above sharp minimum rare cases, momentum based optimization methods will obviously dampen the updated amount at each iteration point, thus *comparing* to traditional gradient descent methods they'll tend to not overshoot at point 6 and beyond.
  
  
  ---
  
  ### Answer by username_4 on 2022-06-21. Score: 0
  
  What your professor most probably means is that since momentum adds your previous gradient to the current (as in a moving average). The speed built up till point 7 (moving in to the right) will added to the gradient from 7 to 8 that is going to the left, causing them to cancel each other out (what might cause the update to turn small enough for the loss to converge instead of explode).
  So it helps at the point where the gradient suddenly changes direction, as the momentum will cancel it out partially.
  
  
  ---
  
  