Answers.Body: <p>If <a href="https://en.wikipedia.org/wiki/Gradient_descent" rel="nofollow noreferrer">gradient descent</a> is like walking down a slope, momentum would be the literal momentum of the agent traversing the <a href="https://en.wikipedia.org/wiki/Hyperplane" rel="nofollow noreferrer">hyperplane</a>. </p>
  
  <p>Under that analogy then, momentum factor would be analogous to the <a href="https://simple.wikipedia.org/wiki/Coefficient_of_friction" rel="nofollow noreferrer">friction coefficient</a>, with 1 being max friction and 0 being no friction.</p>
  
  <p>You should be able to see why there can't be friction beyond that range: if friction = 1 it would be identical to having no friction; if friction &lt;= 0 then by conservation of energy gradient descent will not find a local <a href="https://en.wikipedia.org/wiki/Maxima_and_minima" rel="nofollow noreferrer">minima</a>; if friction > 1 then gradient descent would be moving backwards.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-05T01:35:40.360Z
Answers.Text: latter should be fine as the term is well understood in the literature.
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-12T19:04:40.900Z
Answers.LastActivityDate: 2018-05-12T22:51:33.523Z
Answers.LastEditDate: 2018-05-12T22:51:33.523Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.Body: <p>Let's talk about gradient decent!</p>
  <h2>Analogy:</h2>
  <p>So you're standing on a mountain side, and you want to get to the lowest part of this mountain. You have a notepad with you.</p>
  <p>Although actual physics-momentum would be a good analogy here, I'm not gonna use it.</p>
  <p>You're somewhere on this mountain side and you figure out which way is down*, and you jump once a couple of meters in that direction. How big one jump is would depend on how steep the hill is (the length of the gradient), and how much extra you push with your feet. The first time you decide to not really push that much with your feet. The SGD momentum, comes here; you write down in your note pad which direction you went, and how far (e.g. south, 4 meters).</p>
  <p><strong>Note:</strong> here the PHYSICAL momentum would represent the length of the gradient.</p>
  <p>You repeat this for some time you come to a place where there are only ways upwards.</p>
  <p>Does this mean you hit the bottom? Not necessarily; you might have gotten stuck in a valley, or &quot;local minima&quot;. You really want to get out of this valley, but all directions are upwards, so which way should you jump?</p>
  <p>You now take out your notebook and notice that you've been jumping south east the last 40 steps, and pretty far. You then reason that it is likely that you want to go south east. So you jump south east with a lot of thrust from your feet: This is the intuition on what momentum does;</p>
  <p>If you have a clear &quot;pattern&quot; of which way is down, then this should also count!</p>
  <p><strong>Note:</strong> the momentum only depends on the previous step, but the previous step depends on the steps before that and so on. This is just an analogy.</p>
  <h2>Maths:</h2>
  <p>For the maths, you just add a term that is the last gradient, times some constant.</p>
  <p><code>Heading(t)=γ Heading(t-1)+η Gradient(t)</code></p>
  <p>Where γ is the momentum factor and η is the learning rate.</p>
  <p><a href="http://ruder.io/optimizing-gradient-descent/index.html#momentum" rel="nofollow noreferrer">Sebsastian Ruders blog on gradient descent</a> is brilliant to learn more details of the maths of it.</p>
  <h2>γ ≷ 1</h2>
  <p>For mathematical conclusions:</p>
  <p><code>Heading(t)=γ Heading(t-1)+η Gradient(t)</code></p>
  <p>γ &gt; 1: From the &quot;expression&quot;, you could infer that this case would generate echos. That the gradient of the previous step would contribute more than the actual gradient. For the upcoming step, this effect would get enhanced, and 10 steps down the road, you're stuck going in one direction.</p>
  <p>γ &lt; 1 makes it &quot;converge&quot; to a &quot;terminal velocity&quot;, if you like. It would depend on the preceding steps less and less instead of more and more.</p>
  <p>These effects are pretty clear in the equation you find at <a href="http://ruder.io/optimizing-gradient-descent/index.html#momentum" rel="nofollow noreferrer">Ruders blog</a></p>
  <p>If your momentum term was greater than one, then the notebook would overcome the actual gradient. After a few steps, you wouldn't even look at the hill; you'd go &quot;I've only gone east so far, so I'll just continue east&quot; with your jumps getting longer and longer. This is not good.</p>
  <h2>In conclusion</h2>
  <p>A high momentum term would lead you in the wrong direction (blow up and always go in the same direction), and/or oscillate around the global minima (making you jump too far).</p>
  <p>Hope it helps :)</p>
  <p>*: Strictly speaking, we're finding the &quot;up&quot; direction and go the opposite way. The &quot;up&quot; direction is the gradient.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-13T21:08:30.650Z
Answers.LastActivityDate: 2018-05-13T21:40:08.723Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Andreas Storvik Strauman
Body: <p>Why is it a bad idea to have a momentum factor greater than 1? What are the mathematical motivations/reasons?</p>
  
Tags: <neural-networks><gradient-descent><hyper-parameters><multilayer-perceptrons><momentum>
Title: Why must the momentum factor be in the range 0-1?
UserDisplayName: Ameba kupiec
ThreadText: # Why must the momentum factor be in the range 0-1?
  
  (Asked by: username_0 on 2018-05-12)
  
  Why is it a bad idea to have a momentum factor greater than 1? What are the mathematical motivations/reasons?
  
  **Tags:** neural-networks, gradient-descent, hyper-parameters, multilayer-perceptrons, momentum
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-05-12. Score: 1
  
  If [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent) is like walking down a slope, momentum would be the literal momentum of the agent traversing the [hyperplane](https://en.wikipedia.org/wiki/Hyperplane).
  
  Under that analogy then, momentum factor would be analogous to the [friction coefficient](https://simple.wikipedia.org/wiki/Coefficient_of_friction), with 1 being max friction and 0 being no friction.
  
  You should be able to see why there can't be friction beyond that range: if friction = 1 it would be identical to having no friction; if friction <= 0 then by conservation of energy gradient descent will not find a local [minima](https://en.wikipedia.org/wiki/Maxima_and_minima); if friction > 1 then gradient descent would be moving backwards.
  
  > **Comments:**
  > **username_1**: latter should be fine as the term is well understood in the literature.  
  
  ---
  
  ### Answer by username_2 on 2018-05-13. Score: 0
  
  Let's talk about gradient decent!
  
  ## Analogy:
  
  So you're standing on a mountain side, and you want to get to the lowest part of this mountain. You have a notepad with you.
  
  Although actual physics-momentum would be a good analogy here, I'm not gonna use it.
  
  You're somewhere on this mountain side and you figure out which way is down\*, and you jump once a couple of meters in that direction. How big one jump is would depend on how steep the hill is (the length of the gradient), and how much extra you push with your feet. The first time you decide to not really push that much with your feet. The SGD momentum, comes here; you write down in your note pad which direction you went, and how far (e.g. south, 4 meters).
  
  **Note:** here the PHYSICAL momentum would represent the length of the gradient.
  
  You repeat this for some time you come to a place where there are only ways upwards.
  
  Does this mean you hit the bottom? Not necessarily; you might have gotten stuck in a valley, or "local minima". You really want to get out of this valley, but all directions are upwards, so which way should you jump?
  
  You now take out your notebook and notice that you've been jumping south east the last 40 steps, and pretty far. You then reason that it is likely that you want to go south east. So you jump south east with a lot of thrust from your feet: This is the intuition on what momentum does;
  
  If you have a clear "pattern" of which way is down, then this should also count!
  
  **Note:** the momentum only depends on the previous step, but the previous step depends on the steps before that and so on. This is just an analogy.
  
  ## Maths:
  
  For the maths, you just add a term that is the last gradient, times some constant.
  
  `Heading(t)=γ Heading(t-1)+η Gradient(t)`
  
  Where γ is the momentum factor and η is the learning rate.
  
  [Sebsastian Ruders blog on gradient descent](http://ruder.io/optimizing-gradient-descent/index.html#momentum) is brilliant to learn more details of the maths of it.
  
  ## γ ≷ 1
  
  For mathematical conclusions:
  
  `Heading(t)=γ Heading(t-1)+η Gradient(t)`
  
  γ > 1: From the "expression", you could infer that this case would generate echos. That the gradient of the previous step would contribute more than the actual gradient. For the upcoming step, this effect would get enhanced, and 10 steps down the road, you're stuck going in one direction.
  
  γ < 1 makes it "converge" to a "terminal velocity", if you like. It would depend on the preceding steps less and less instead of more and more.
  
  These effects are pretty clear in the equation you find at [Ruders blog](http://ruder.io/optimizing-gradient-descent/index.html#momentum)
  
  If your momentum term was greater than one, then the notebook would overcome the actual gradient. After a few steps, you wouldn't even look at the hill; you'd go "I've only gone east so far, so I'll just continue east" with your jumps getting longer and longer. This is not good.
  
  ## In conclusion
  
  A high momentum term would lead you in the wrong direction (blow up and always go in the same direction), and/or oscillate around the global minima (making you jump too far).
  
  Hope it helps :)
  
  \*: Strictly speaking, we're finding the "up" direction and go the opposite way. The "up" direction is the gradient.
  
  
  ---
  
  