Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes "over adapted". We design a new algorithm, called Partially adaptive momentum estimation method, which unifies the Adam/Amsgrad with SGD by introducing a partial adaptive parameter p, to achieve the best from both worlds. We also prove the convergence rate of our proposed algorithm to a stationary point in the stochastic nonconvex optimization setting. Experiments on standard benchmarks show that our proposed algorithm can maintain fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks. These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.
% post author response %
Thanks for your detailed response. 
R1. Note that in almost all classical optimization routines, the learning rate has a (very intuitive) scaling on the problem parameters - for e.g. in gradient descent, the learning rate looks like 1/smoothness. This is mirrored in the definition of Newton methods - where, in the direction of hessian^{-1} grad, one uses a scale free line search to estimate an appropriate stepsize between 0 and 1 (to emphasize, this value has *no* dependence on problem scaling). While this begins to fall apart with the case of adaptive gradient methods, how can we even hope to justify the potentially arbitrary power of, say, 1/4 or 1/8 used by Padam? This is the reason behind my comment that the algorithm is unnatural. By using such a power of the smoothness of the problem, the other component of the learning rate (alpha_t) is no longer a scale free quantity. It has to depend on other problem dependent parameters for the overall learning rate to be scaled appropriately based on the problem characteristics. 
R2. While the paper performs grid search (sec C.3 in the paper) for the partially adaptive parameter, the lambda value for YOGI is set as one suggested in their paper. I can accept the claim of authors if I see more experiments tuning the lambda parameter for YOGI as well.  Otherwise, I dont quite see why one specific parameter value for lambda works for every problem.
R3/R5. My point is that the original paper for adaptive methods (adam/adagrad) never mentions learning rate decay. This seems to have been added in subsequently just to boost the performance. What I do not understand is what is the specific advantage of adaptive methods over SGD if every component used by SGD (including learning rate decay) is used even by adaptive methods. Even from a theoretical bound perspective, to the best of my knowledge, there is no clear indication that the partially adaptive momentum methods actually improve over vanilla sgd (+momentum) in settings that do not involve dense parameters/gradients.
R4. Again, the learning rate decay has a specific use in the papers I referenced in my review. Somehow, the bounds presented in the paper do not reflect the use of a step decay schedule on the learning rates, so, I see that the theorem (aside from the assumptions mentioned) is detached from the practical results even in this respect. 
%%
This paper considers generalization issues experienced by adaptive gradient methods compared to well-tuned SGD + momentum, a topic that is of interest in the development of optimization methods for deep learning. The paper is well-written and elaborates on (i) issues faced by adaptive gradient methods in contrast to standard SGD + momentum, (ii) presents experimental results on training standard conv-net based architectures on image classification benchmarks and on training LSTMs on PTB and (iii) presenting theoretical analysis relating convergence of the method to a first-order critical point for smooth stochastic non-convex optimization.
I have questions about certain aspects of the paper, which I will elaborate below:
— If one takes a step back to understand the origins of diagonal adaptation methods (introduced by Adagrad), this was motivated by the infeasibility of using the inverse square root of a full pre-conditioning adaptation matrix. If we think of such full matrix adaptation methods, is this paper implying the use of other matrix powers (other than a square root) as used by adagrad (or other adaptation approaches)? This appears very unnatural to me.
-- If the main issue is that it is not possible to typically use a large learning rate at the start with ADAM or other preconditioning methods and that leads to using other powers of the diagonal adaptation matrix, a more natural fix would be to use a conservative (trust region inspired) approach to reduce aggressive steps at the start. What I mean is as follows: Suppose H_t is a preconditioning matrix, g_t is the gradient (or the discounted sum of gradients with/without bias correction). The current approach is to make H_t diagonal and use H_t ^{-1/2}g_t as the update matrix. One option to prevent aggressive initial steps is to use (H_t + \lambda)^{-1/2} g_t, either with having lambda fixed through the optimization or making it a function of iterations. 
— I am not aware that papers which employ adaptive methods also tend to use some form of learning rate decay (on the alpha_t ’s) - at least, by looking at the original papers of Adam/adagrad, I do not see the combination of learning rate decay and adaptive methods. In a sense, if one has an `"adaptive" optimization method, it’d be unnatural to have to use some form of step decay of the learning rates (alpha_t's) in conjunction with these methods. One would typically just use SGD+momentum with some form of such a step decay of the learning rates. This to me is a serious shortcoming - it appears to make the use of adaptive methods almost irrelevant because the only hyper-parameter that it gets rid of is the dependence on the initial learning rate (because, for SGD, we anyway have the other hyper-parameters like momentum, stepdecay factor, when to decay learning rate etc.)
— With regards to theory (and connections to experiments): Despite the fact that the paper employs a step decay schedule on the learning rates (alpha_t), their theorem statement (or any corollary) doesn’t actually employ this specific step size decay scheme (on the alpha_t ’s) and attempts to understand what are the advantages of the step decay schedule on the convergence statements provided.  The step decay schedule has featured in several recent efforts in the stochastic optimization community (both with convex (https://arxiv.org/pdf/1607.01027.pdf, https://arxiv.org/pdf/1904.12838.pdf)/non-convex (e.g. https://arxiv.org/pdf/1907.09547.pdf) objectives), where, the results indicate non-trivial advantages of using these step-decay schemes on alpha_t’s (though, these are with non-adaptive optimization methods).
— Along these lines (of the previous point), it is important to note what the performance of the optimization method is when alpha_t’s are fixed to a specific value (without being decayed over the course of optimization) - since this relates to the most standard definition (and advantage) associated with using adaptive gradient methods. My understanding is that this result continues to be fairly sub-optimal compared to using SGD+momentum with a step decay schedule. 
Other minor comments:
— I do not understand the use of the term “second order” momentum for calling variables that have a running average of squared gradients. This term is misleading in what it represents.