Body: <p>I've been working on <a href="https://arxiv.org/abs/1708.05031" rel="nofollow noreferrer">Neural Collaborative Filtering (NCF)</a> recently to build a recommender system using <a href="https://www.tensorflow.org/recommenders" rel="nofollow noreferrer">Tensorflow Recommenders</a>. Doing some hyperparameter tuning with different optimizers available in the module <a href="https://www.tensorflow.org/api_docs/python/tf/keras/optimizers" rel="nofollow noreferrer">tf.keras.optimizers</a>, I found out that Adam and its other variants, such as Adamax and Nadam, work much slower than seemingly less advanced optimizers, like Adagrad, Adadelta, and SGD. With Adam and its variants, training each epoch takes about 30x longer.</p>
  <p>It came out as a surprise to me, knowing one of the most cherished properties of Adam optimizer is its convergence speed, especially compared to SGD. What could be the reason for such a significant difference in computation speed?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-06T17:59:53.010Z
Comments.Text: Just to make sure I understand your question, are you asking why the time taken to perform 1 epoch with Adam is longer than the time used to perform 1 epoch with SGD, in the context of NCF?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-07T07:48:35.333Z
Comments.Text: Exactly. In my case, it is clear that Adam or other Adam-like optimizers converge faster in terms of the number of epochs that it takes them to reach a better set of parameters. However, it takes much longer for them to complete one epoch. Therefore it ends up taking much longer to train the network using such optimizers. I hope this clarifies your question.
Comments.UserDisplayName: bkaankuguoglu
Tags: <neural-networks><deep-learning><training><hyperparameter-optimization><optimizers>
Title: Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?
UserDisplayName: bkaankuguoglu
ThreadText: # Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?
  
  (Asked by: username_0 on 2021-05-06)
  
  I've been working on [Neural Collaborative Filtering (NCF)](https://arxiv.org/abs/1708.05031) recently to build a recommender system using [Tensorflow Recommenders](https://www.tensorflow.org/recommenders). Doing some hyperparameter tuning with different optimizers available in the module [tf.keras.optimizers](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers), I found out that Adam and its other variants, such as Adamax and Nadam, work much slower than seemingly less advanced optimizers, like Adagrad, Adadelta, and SGD. With Adam and its variants, training each epoch takes about 30x longer.
  
  It came out as a surprise to me, knowing one of the most cherished properties of Adam optimizer is its convergence speed, especially compared to SGD. What could be the reason for such a significant difference in computation speed?
  
  **Tags:** neural-networks, deep-learning, training, hyperparameter-optimization, optimizers
  
  > **Comments:**
  > **username_1**: Just to make sure I understand your question, are you asking why the time taken to perform 1 epoch with Adam is longer than the time used to perform 1 epoch with SGD, in the context of NCF?  
  > **username_0**: Exactly. In my case, it is clear that Adam or other Adam-like optimizers converge faster in terms of the number of epochs that it takes them to reach a better set of parameters. However, it takes much longer for them to complete one epoch. Therefore it ends up taking much longer to train the network using such optimizers. I hope this clarifies your question.  
  
  ---
  
  ## 0 answers
  
  