First-order adaptive optimization algorithms such as Adam play an important role in modern deep learning due to their super fast convergence speed in solving large scale optimization problems. However, Adam's non-convergence behavior and disturbing generalization ability make it fall into a love-hate relationship to the deep learning community. Previous studies on Adam and its variants (refer as Adam-Type algorithms) mainly rely on theoretical regret bound analysis, which overlooks the natural characteristics residing in Adam-Type algorithms and limits our understanding. In this paper, we aim at seeking a different interpretation of Adam-Type algorithms so that we can intuitively comprehend and improve them. The way we chose is based on a traditional online convex optimization algorithm scheme known as mirror descent method. By bridging Adam and mirror descent algorithm, we receive a clear map of the functionality of each term in Adam. In addition, this new angle brings us a new insight on identifying the non-convergence issue of Adam and explaining the superior convergence speed of Adam than other first-order methods. Moreover, we provide a new variant of Adam-Type algorithm, namely AdamAL which can naturally mitigate the non-convergence issue of Adam and improve its performance. We further conduct experiments on various popular deep learning tasks and models, and the results are quite promising.
Summary:
This work proposed a framework to analyze both Adam-type algorithms and SGD-type algorithms. The authors considered both of them as specialized cases of mirror descent algorithms and provided a new algorithm AdamAL. The authors showed experiments to backup their theoretical results. 
Pros:
The authors provided a novel framework to analyze Adam-type algorithms by using standard FTRL framework. It provides a unified viewpoint to consider a broad class of algorithms. The authors also provided a new algorithm AdamAL to overcome some shortcomings in previous algorithms. 
Cons:
- The novelty of this paper is limited. The authors provided a framework to analyze Adam-type algorithms. However, it seems that contribution is more conceptual rather than practical. I suggest the authors add more examples or theorems to show the superiority of their mirror descent framework. 
- Some of the explanations in this paper may be wrong. For instance, around equation 17, the authors suggested that if the swap happens, then v_{t+1} = v_t. That is not correct since the swap happens for each coordinate. Meanwhile, the explanation about why AdamAL is better than AMSgrad is quite poor since the update rule of AMSgrad can also guarantee the coordinate decreasing of v_t. I suggest the authors explain more on the algorithm design. 
- The authors should complete the proof of Theorem 3.1. 
- The settings of experiments are limited. The authors should at least compare AdamAL with other baseline algorithms on some modern deep learning tasks including Imagenet.  
Minor comments:
- Below equation 8, detail -> details.
- The authors should add the definition of 1:t in subscript for g_{1:t} or \phi_{1:t}.
- Page 6, the first paragraph, logt-> \log t
- This paper lacks some references in this area. 
J. Chen and Q. Gu. Closing the generalization gap of adaptive gradient methods in training
deep neural networks. arXiv preprint arXiv:1806.06763, 2018.
Ward, R., Wu, X. and Bottou, L. (2018). Adagrad stepsizes: Sharp convergence over nonconvex
landscapes, from any initialization. arXiv preprint arXiv:1806.01811 .
Li, X. and Orabona, F. (2018). On the convergence of stochastic gradient descent with adaptive
stepsizes. arXiv preprint arXiv:1805.08114 .