Body: <p>AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy.</p>
  <p>I've recently learned that it is not as helpful for RNNs, where plain SGD is preferred.</p>
  <p>For example, Deep Learning by Goodfellow et. al says (<a href="https://www.deeplearningbook.org/contents/rnn.html" rel="nofollow noreferrer">section 10.11, page 401</a>):</p>
  <blockquote>
  <p>Both of these approaches have largely been replaced by simply using SGD (even without momentum) applied to LSTMs.</p>
  </blockquote>
  <p>The author talks about LSTMs and &quot;both of these approaches&quot; refer to second-order and first-order SGD methods with momentum methods, respectively, according to my understanding.</p>
  <p>What causes this discrepancy?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-09T05:52:34.193Z
Comments.Text: The momentum does not always help on CNN either
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-09T05:55:13.867Z
Comments.Text: @mirror2image Sure, but I suppose it's less common to use it for RNNs than CNNs.
Comments.UserDisplayName: SpiderRico
Tags: <recurrent-neural-networks><gradient-descent><stochastic-gradient-descent><momentum>
Title: Why do momentum techniques not work well for RNNs?
UserDisplayName: SpiderRico
ThreadText: # Why do momentum techniques not work well for RNNs?
  
  (Asked by: username_0 on 2020-03-09)
  
  AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy.
  
  I've recently learned that it is not as helpful for RNNs, where plain SGD is preferred.
  
  For example, Deep Learning by Goodfellow et. al says ([section 10.11, page 401](https://www.deeplearningbook.org/contents/rnn.html)):
  
  > Both of these approaches have largely been replaced by simply using SGD (even without momentum) applied to LSTMs.
  
  The author talks about LSTMs and "both of these approaches" refer to second-order and first-order SGD methods with momentum methods, respectively, according to my understanding.
  
  What causes this discrepancy?
  
  **Tags:** recurrent-neural-networks, gradient-descent, stochastic-gradient-descent, momentum
  
  > **Comments:**
  > **username_1**: The momentum does not always help on CNN either  
  > **username_0**: @username_1 Sure, but I suppose it's less common to use it for RNNs than CNNs.  
  
  ---
  
  ## 0 answers
  
  