A common belief in the machine learning community is that using adaptive gradient methods hurts generalization. We re-examine this belief both theoretically and experimentally, in light of insights and trends from recent years. We revisit some previous oft-cited experiments and theoretical accounts in more depth, and provide a new set of experiments in larger-scale, state-of-the-art settings. We conclude that with proper tuning, the improved training performance of adaptive optimizers does not in general carry an overfitting penalty, especially in contemporary deep learning. Finally, we synthesize a "user's guide" to adaptive optimizers, including some proposed modifications to AdaGrad to mitigate some of its empirical shortcomings.
This paper revisited the a common belief that adaptive gradient methods hurts generalization performances. The authors re-examine this in more depth and provide a new set of experiments in larger-scale, state-of-the-art settings. The authors claimed that with proper tuning, the performance of adaptive optimizers can mitigate the gap  with non-adaptive methods.
- It is great to have someone revisit and challenge the conventional ideas in the community. However, I did not find much insightful information in this paper. As the author mentioned, there are many recent works focusing on further improving the empirical generalization performances of adaptive gradient methods. The main aspects mentioned in the paper, \epsilon tuning, learning rate warmup and decaying schedule, are not something new and many of which are mentioned or used in the recent advances. This makes the contribution of this paper look like combining all the tricks together. The authors might want to carefully comment the differences with the recent advances
[1] Liu, Liyuan, et al. "On the variance of the adaptive learning rate and beyond." arXiv preprint arXiv:1908.03265 (2019).
[2] Loshchilov, Ilya, and Frank Hutter. "Decoupled weight decay regularization." ICLR 2019.
[3] Zaheer, Manzil, et al. "Adaptive methods for nonconvex optimization." Advances in Neural Information Processing Systems. 2018.
[4] Chen, Jinghui, and Quanquan Gu. "Closing the generalization gap of adaptive gradient methods in training deep neural networks." arXiv preprint arXiv:1806.06763 (2018).
[5] Luo, Liangchen, et al. "Adaptive gradient methods with dynamic bound of learning rate." arXiv preprint arXiv:1902.09843 (2019).
- In terms of tuning \epsilon, the authors mentioned that default setting in Tensorflow is 0.1 for Adagrad which is too high. However, most papers regarding adaptive gradient method usually set \epsilon as 10^-8. Pytorch set default value as 10^-10. In fact, Yogi paper mentioned above gives some different conclusions. In their experiments, they found that setting \epsilon to be a bit larger like 10^-3 give better results compared with 10^-8. I wonder if the authors examine the reasons for different conclusions here?
 
- at the end of page 6, missing reference for histogram