Answers.Body: <p>The book <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer">Deep Learning</a> by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that</p>
  
  <blockquote>
    <p>Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the rate of convergence.</p>
  </blockquote>
  
  <p>I'm not sure why this is, but the theoretical advantage depends on a convex problem, and from this, it sounds like the practical advantage does too - or at least, that it isn't applicable to typical neural network landscapes.</p>
  
  <p>Perhaps it can be implemented more efficiently, but it seems to me that you need to do parameter updates twice (in order to calculate the gradient in a place you aren't moving two) and thus Nesterov requires more computation and memory than plain ole momentum.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-06T12:41:33.863Z
Answers.Text: It depends what you mean by "improving" convergence - finding a better optimum or getting there faster.  I think you mean the former (and are mostly correct, but there may be exceptions which are probably hard to quantify), while Goodfellow et al mean the latter (the following discussion discusses rate of convergence).
Answers.UserDisplayName: Ketil Malde
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-01T22:22:25.617Z
Answers.Text: It's frustrating that the book makes this claim without any citation.
Answers.UserDisplayName: japreiss
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-09T22:19:13.463Z
Answers.Text: Let $y_t = x_t + \beta_t (x_t - x_{t-1})$.
  Nesterov momentum is $x_{t+1} = y_t - \alpha_t f'(y_t)$.
  Classical momentum is $x_{t+1} = y_t - \alpha_t f'(x_t)$.
  Isn't that the same number of additions/subtractions?
Answers.UserDisplayName: user76284
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-05T06:48:35.293Z
Answers.LastActivityDate: 2020-02-05T06:48:35.293Z
Answers.UserDisplayName: Ketil Malde
Body: <p>I've been reading about Nesterov momentum from <a href="https://ruder.io/optimizing-gradient-descent/index.html#gradientdescentvariants" rel="nofollow noreferrer">here</a> and it seems like a nice improvement over regular momentum with no extra cost whatsoever.</p>
  
  <p>However, is this really the case? Are there instances where regular momentum performs better than Nesterov momentum or is Nesterov momentum performs at least as good as the regular momentum all the time?</p>
  
Tags: <neural-networks><comparison><objective-functions><gradient-descent><hyperparameter-optimization>
Title: Is there a reason to choose regular momentum over Nesterov momentum for neural networks?
UserDisplayName: SpiderRico
ThreadText: # Is there a reason to choose regular momentum over Nesterov momentum for neural networks?
  
  (Asked by: username_0 on 2020-02-04)
  
  I've been reading about Nesterov momentum from [here](https://ruder.io/optimizing-gradient-descent/index.html#gradientdescentvariants) and it seems like a nice improvement over regular momentum with no extra cost whatsoever.
  
  However, is this really the case? Are there instances where regular momentum performs better than Nesterov momentum or is Nesterov momentum performs at least as good as the regular momentum all the time?
  
  **Tags:** neural-networks, comparison, objective-functions, gradient-descent, hyperparameter-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-05. Score: 4 | Accepted answer
  
  The book [Deep Learning](https://www.deeplearningbook.org/) by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that
  
  > Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the rate of convergence.
  
  I'm not sure why this is, but the theoretical advantage depends on a convex problem, and from this, it sounds like the practical advantage does too - or at least, that it isn't applicable to typical neural network landscapes.
  
  Perhaps it can be implemented more efficiently, but it seems to me that you need to do parameter updates twice (in order to calculate the gradient in a place you aren't moving two) and thus Nesterov requires more computation and memory than plain ole momentum.
  
  > **Comments:**
  > **username_1**: It depends what you mean by "improving" convergence - finding a better optimum or getting there faster.  I think you mean the former (and are mostly correct, but there may be exceptions which are probably hard to quantify), while Goodfellow et al mean the latter (the following discussion discusses rate of convergence).  
  > **username_2**: It's frustrating that the book makes this claim without any citation.  
  > **username_3**: Let $y_t = x_t + \beta_t (x_t - x_{t-1})$. Nesterov momentum is $x_{t+1} = y_t - \alpha_t f'(y_t)$. Classical momentum is $x_{t+1} = y_t - \alpha_t f'(x_t)$. Isn't that the same number of additions/subtractions?  
  
  ---
  
  