Adaptive gradient methods have experienced great success in training deep neural networks (DNNs). The basic idea of the methods is to track and properly make use of the first and/or second moments of the gradient for model-parameter updates over iterations for the purpose of removing the need for manual interference. In this work, we propose a new adaptive gradient method, referred to as generalized adaptive moment estimation (Game). From a high level perspective, the new method introduces two more parameters w.r.t. AMSGrad (S. J. Reddi & Kumar (2018)) and one more parameter w.r.t. PAdam (Chen & Gu (2018)) to enlarge the parameterselection space for performance enhancement while reducing the memory cost per iteration compared to AMSGrad and PAdam. The saved memory space amounts to the number of model parameters, which is significant for large-scale DNNs. Our motivation for introducing additional parameters in Game is to provide algorithmic flexibility to facilitate a reduction of the performance gap between training and validation datasets when training a DNN. Convergence analysis is provided for applying Game to solve both convex optimization and smooth nonconvex optimization. Empirical studies for training four convolutional neural networks over MNIST and CIFAR10 show that under proper parameter selection, Game produces promising validation performance as compared to AMSGrad and PAdam.
Summary
------
The authors propose an adaptation of the Adam method, with the AMSGrad correction and an additional parameter to p to exponentiate the diagonal conditioning matrix V (Padam).
The proposed method changes two aspects: first, there is no need to retain two version of the rescaling matrix v, where amsgrad and Padam keeps the last monotone \hat v)t and non-monotone version v_t. Secondly, a new parameter q is introduced, that replaces the q=2 in the moment estimation phase of (P)Adam.
A regret analysis is proposed in the convex case, while a vanishing bound on the gradient is derived in the non-convex smooth case.
Review
------
Although improving optimization methods is certainly important for the machine learning community, the reviewer have strong concerns about this paper.
First of all, the paper is hard to read as it contains too many approximations. What does 'SGD is known to work reasonably well regardless of their problem structure' means ? Same thing for 'Its performance deteriorates when the gradients are dense due to a rapid decay of the learning rates.' The authors uses many times elliptical discourse to detail the course of their analysis, which is non informative: for instance, 'one can easily derive the upper bound expression', and 'It is not difficult to conclude that when G_t [...]'. This level of writing is not professional. Some completely irrelevant argument are proposed to justify the method: 'For instance the extension from l_2 norm to l_p norm and generalization from Cauchy-Schwart to Holder inequality'.
The reviewer has interrogations about the relevance of the proposed algorithm. The additional parameter q needs to be tuned, which carries only the promise of further overfitting. I would have been convinced by an sequence of experiment where q is set automatically by considering a validation set, and then tested on a left out test set. However, the authors report only the results for the best q, with non significant differences (and not quantified, there is no result tables). Using q=2 at least made sense from the point of view of empirical Fisher matrix approximation.
The review also have several concerns aout the correctness of the proposed arguments. First, the major argument of memory usage stems from 1) a miscalculation and 2) a misunderstanding of memory bottlenecks in deep learning. 1) adam models keeps in memory x_t (the model parameter), g_t (the model gradient), m_t (momentum) \hat v_t and v_{t-1} (the monotone and non monotone version of the second order moment estimation. In contrast, the proposed model do not track v_{t-1}: this amounts to a memory saving of 20% considering all model related parameters. 2) more importantly, the most important memory usage in deep leaning comes from the activations that need to be kept in memory during the forward pass to perform the backward pass. Even the biggest model are less than 1GB, and most of the memory used during training is dedicated to intermediary activations. This makes the major argument of the paper less convincing, and misleads the reader.
Second, even when disregarding the slightly abusive assumptions over the iterate sequences, that are common in the adaptive stochastic optimizers community, I think that the bound proposed in theorem 1 is non informative, as the second term behaves like T sqrt(T) assymptotically, due to the presence of 1 / \alpha_t. This does not show the convergence of averaged regret R_T / T.
Regarding the experiment section, I am afraid that testing a new optimizer over MNIST and CIFAR is not enough to show the relevance of the method for the whole deep learning community. An eperiment over a non-toy dataset (eg ImageNet), and on non computer-vision dataset (eg from NLP) would be a minimum, besides the overfitting concern described above.
In conclusion, it is the reviewer's opinion that significant rework in term of presentation and strong improvement of the experiment section to make the case for the Game optimizer.
Minor comments
------------
Table 1: what do you bound when you compare results ? I think there is a typo in Zhou et al. result: 1/2 should read p.
Eq (1): it is rather surprising to use x_t as the model parameters in the ICLR community. 
p 7: the dimension d could be larger than T when training large-scale neural networks: how does it relate to comparing sqrt(dT) to (dT)^s ?