Answers.Body: <p>Empirically, I observed that algorithms like Adam and RMSProp tended to give me a final higher performance (in my case, the accuracy) on (the validation dataset) with respect to SGD. However, I also observed that Adam and RMSProp are highly sensitive to certain values of the learning rate (and, sometimes, other hyper-parameters like the batch size) and they can catastrophically fail to converge if e.g. the learning rate is too high. On the other hand, in general, SGD have not led me to the highest performance, but they did not catastrophically fail (at least, as much as Adam and RMSProp) in my experiments (even when using quite different hyper-parameters). I noticed that the learning rate (and the batch size) are the hyper-parameters that mainly affect the performance of all these algorithms.</p>
  
  <p>In my experiments, I use SGD without momentum and I used the (PyTorch) default values of Adam and RMSProp. I only compared SGD with Adam and RMSProp, on the relatively simple task of recognising MNIST digits. You can have a look at this repository <a href="https://github.com/nbro/comparative-study-between-optimizers" rel="noreferrer">https://github.com/nbro/comparative-study-between-optimizers</a>, which contains the code I used to perform these experiments. You also have the instructions there to perform the experiments (if you want).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-26T13:58:01.257Z
Answers.Text: I will check out the GitHub repository! Thank you for your answer.
Answers.UserDisplayName: Utku
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-26T14:38:32.890Z
Answers.Text: The accuracy stayed low (like $12 \%$) all the time.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-26T15:00:48.247Z
Answers.Text: Yeah, it was about recognising 1 of the 10 MNIST digits.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-26T10:01:46.080Z
Answers.LastActivityDate: 2019-03-26T10:01:46.080Z
Answers.UserDisplayName: nbro
Body: <p>As far as I know, Stochastic Gradient Descent is an optimization algorithm which belongs to the the category of algorithms where hyper-parameters have to be defined beforehand. They are useful in many cases, but there are some cases that the adaptive learning algorithms (like AdaGrad or Adam) might be preferable. </p>
  
  <p>When are algorithms like Adam and AdaGrad preferred over SGD? What are the cons and pros of adaptive algorithms, like Adam, when we compare them with learning algorithms like SGD?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-26T20:06:14.283Z
Comments.Text: By the way, your understanding of the difference between SGD and algorithms like Adam is incorrect. In Adam, you also need to specify certain hyper-parameters beforehand. Read the paper [An overview of gradient descent optimization algorithms](https://arxiv.org/pdf/1609.04747.pdf) for a gentle overview of optimization algorithms in ML.
Comments.UserDisplayName: nbro
Tags: <machine-learning><optimization>
Title: When should we use algorithms like Adam as opposed to SGD?
UserDisplayName: Utku
ThreadText: # When should we use algorithms like Adam as opposed to SGD?
  
  (Asked by: username_0 on 2019-03-25)
  
  As far as I know, Stochastic Gradient Descent is an optimization algorithm which belongs to the the category of algorithms where hyper-parameters have to be defined beforehand. They are useful in many cases, but there are some cases that the adaptive learning algorithms (like AdaGrad or Adam) might be preferable.
  
  When are algorithms like Adam and AdaGrad preferred over SGD? What are the cons and pros of adaptive algorithms, like Adam, when we compare them with learning algorithms like SGD?
  
  **Tags:** machine-learning, optimization
  
  > **Comments:**
  > **username_1**: By the way, your understanding of the difference between SGD and algorithms like Adam is incorrect. In Adam, you also need to specify certain hyper-parameters beforehand. Read the paper [An overview of gradient descent optimization algorithms](https://arxiv.org/pdf/1609.04747.pdf) for a gentle overview of optimization algorithms in ML.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-26. Score: 6 | Accepted answer
  
  Empirically, I observed that algorithms like Adam and RMSProp tended to give me a final higher performance (in my case, the accuracy) on (the validation dataset) with respect to SGD. However, I also observed that Adam and RMSProp are highly sensitive to certain values of the learning rate (and, sometimes, other hyper-parameters like the batch size) and they can catastrophically fail to converge if e.g. the learning rate is too high. On the other hand, in general, SGD have not led me to the highest performance, but they did not catastrophically fail (at least, as much as Adam and RMSProp) in my experiments (even when using quite different hyper-parameters). I noticed that the learning rate (and the batch size) are the hyper-parameters that mainly affect the performance of all these algorithms.
  
  In my experiments, I use SGD without momentum and I used the (PyTorch) default values of Adam and RMSProp. I only compared SGD with Adam and RMSProp, on the relatively simple task of recognising MNIST digits. You can have a look at this repository <https://github.com/username_1/comparative-study-between-optimizers>, which contains the code I used to perform these experiments. You also have the instructions there to perform the experiments (if you want).
  
  > **Comments:**
  > **username_0**: I will check out the GitHub repository! Thank you for your answer.  
  > **username_1**: The accuracy stayed low (like $12 \%$) all the time.  
  > **username_1**: Yeah, it was about recognising 1 of the 10 MNIST digits.  
  
  ---
  
  