Answers.Body: <p>Since they're various kinds of the same idea of leveraging exponentially weighted average of the gradients, they may not have <em>significant</em> difference albeit each is most suitable for different cases.</p>
  <p>Nowadays the usually standard definition of momentum is either your <a href="https://optimization.cbe.cornell.edu/index.php?title=Momentum" rel="nofollow noreferrer">1st</a> or <a href="https://optimization.cbe.cornell.edu/index.php?title=Adam" rel="nofollow noreferrer">3rd</a> definition as referenced, and thus they're most similar, though obviously the primary difference is the scaling factor <span class="math-container">$(1−)$</span> in the 1st definition which helps in further reducing the contribution of the current gradient versus the past accumulated gradients for the momentum. This makes the 1st definition a bit more stable and less sensitive to the choice of <span class="math-container">$β$</span> which in practice is set as <span class="math-container">$0.9$</span> in most libraries and it's used in the popular and more adaptive Adam algo considering the propagation of root mean squared gradients as referenced above.</p>
  <p>Your <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Momentum" rel="nofollow noreferrer">2nd</a> definition as referenced is the original momentum formulation of Polyak's 1964 article on solving functional equations and can be easily converted to a form similar to your 3rd definition, ie, <span class="math-container">$\theta=\theta - \alpha v_{\theta_t}$</span>, where <span class="math-container">$v_{\theta_t} = -\frac{\beta}{\alpha} v_{\theta_{t-1}} + ∂\theta_t$</span>. Therefore once you define a new <span class="math-container">$\beta$</span> to replace <span class="math-container">$-\frac{\beta}{\alpha}$</span>, it's equivalent to your 3rd definition.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-06T21:41:56.297Z
Answers.LastActivityDate: 2024-07-06T21:41:56.297Z
Answers.UserDisplayName: cinch
Body: <p>I'm seeing different manners to define momentum, I'm not sure if there is significant difference or not.</p>
  <p>From my thinking, they seem to do a similar thing mathematically and in practice but I'm curious to know if there's a significant difference that i'm not seeing.</p>
  <h2><strong>1st Definition</strong></h2>
  <p><strong>Momentum:</strong>
  <span class="math-container">$v\theta_{t} = \beta * v\theta_{t-1} + (1 - \beta) * ∂\theta_t$</span></p>
  <p><strong>Update</strong>: <span class="math-container">$\theta = \theta - \alpha * v\theta$</span></p>
  <h2><strong>2nd Definition</strong></h2>
  <p><strong>Momentum:</strong> <span class="math-container">$v\theta_t = \beta * v\theta_{t-1} - \alpha *∂\theta_t$</span></p>
  <p><strong>Update</strong>: <span class="math-container">$\theta = \theta + v\theta_t$</span></p>
  <h2>3rd Definition</h2>
  <p><strong>Momentum:</strong> <span class="math-container">$v\theta_t = \beta * v\theta_{t-1} + ∂\theta_t$</span></p>
  <p><strong>Update:</strong> <span class="math-container">$\theta = \theta - \alpha * v\theta_t$</span></p>
  <p>From my understanding, the first definition explicitly uses the exponentially weighted average with the <span class="math-container">$\beta$</span> hyperparameter, which then can increase the stability of the model when using momentum and makes <span class="math-container">$\beta$</span> more intuitive to tune.</p>
  <p>I'm not sure why I'd want to use the 2nd or 3rd definitions over the 1st, nor if they have any significant difference.</p>
  
Tags: <optimization><gradient-descent><momentum>
Title: Different Definitions of Momentum -- which one should I work with?
UserDisplayName: vxnuaj
ThreadText: # Different Definitions of Momentum -- which one should I work with?
  
  (Asked by: username_0 on 2024-07-03)
  
  I'm seeing different manners to define momentum, I'm not sure if there is significant difference or not.
  
  From my thinking, they seem to do a similar thing mathematically and in practice but I'm curious to know if there's a significant difference that i'm not seeing.
  
  ## **1st Definition**
  
  **Momentum:**
  $v\theta\_{t} = \beta \* v\theta\_{t-1} + (1 - \beta) \* ∂\theta\_t$
  
  **Update**: $\theta = \theta - \alpha \* v\theta$
  
  ## **2nd Definition**
  
  **Momentum:** $v\theta\_t = \beta \* v\theta\_{t-1} - \alpha \*∂\theta\_t$
  
  **Update**: $\theta = \theta + v\theta\_t$
  
  ## 3rd Definition
  
  **Momentum:** $v\theta\_t = \beta \* v\theta\_{t-1} + ∂\theta\_t$
  
  **Update:** $\theta = \theta - \alpha \* v\theta\_t$
  
  From my understanding, the first definition explicitly uses the exponentially weighted average with the $\beta$ hyperparameter, which then can increase the stability of the model when using momentum and makes $\beta$ more intuitive to tune.
  
  I'm not sure why I'd want to use the 2nd or 3rd definitions over the 1st, nor if they have any significant difference.
  
  **Tags:** optimization, gradient-descent, momentum
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-06. Score: 0 | Accepted answer
  
  Since they're various kinds of the same idea of leveraging exponentially weighted average of the gradients, they may not have *significant* difference albeit each is most suitable for different cases.
  
  Nowadays the usually standard definition of momentum is either your [1st](https://optimization.cbe.cornell.edu/index.php?title=Momentum) or [3rd](https://optimization.cbe.cornell.edu/index.php?title=Adam) definition as referenced, and thus they're most similar, though obviously the primary difference is the scaling factor $(1−)$ in the 1st definition which helps in further reducing the contribution of the current gradient versus the past accumulated gradients for the momentum. This makes the 1st definition a bit more stable and less sensitive to the choice of $β$ which in practice is set as $0.9$ in most libraries and it's used in the popular and more adaptive Adam algo considering the propagation of root mean squared gradients as referenced above.
  
  Your [2nd](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Momentum) definition as referenced is the original momentum formulation of Polyak's 1964 article on solving functional equations and can be easily converted to a form similar to your 3rd definition, ie, $\theta=\theta - \alpha v\_{\theta\_t}$, where $v\_{\theta\_t} = -\frac{\beta}{\alpha} v\_{\theta\_{t-1}} + ∂\theta\_t$. Therefore once you define a new $\beta$ to replace $-\frac{\beta}{\alpha}$, it's equivalent to your 3rd definition.
  
  
  ---
  
  