Adaptive optimization algorithms such as RMSProp and Adam have fast convergence and smooth learning process. Despite their successes, they are proven to have non-convergence issue even in convex optimization problems as well as weak performance compared with the first order gradient methods such as stochastic gradient descent (SGD). Several other algorithms, for example AMSGrad and AdaShift, have been proposed to alleviate these issues but only minor effect has been observed. This paper further analyzes the performance of such algorithms in a non-convex setting by extending their non-convergence issue into a simple nonconvex case and show that Adam's design of update steps would possibly lead the algorithm to local minimums. To address the above problems, we propose a novel adaptive gradient descent algorithm, named AdaX, which accumulates the long-term past gradient information exponentially. We prove the convergence of AdaX in both convex and non-convex settings. Extensive experiments show that AdaX outperforms Adam in various tasks of computer vision and natural language processing and can catch up with SGD.
This paper introduces a new step-size adaptation algorithm called AdaX. AdaX 
builds on the ideas of the Adam algorithm to address instability and non-convergence issues. 
Convergence of AdaX is proven in both convex and non-convex settings. The paper 
also provides an empirical comparison of AdaX against its predecessors 
(SGD, RMSProp, Adam, AMSGrad) on a variety of tasks.
I recommend the paper be rejected. I believe the convergence results could be a significant contribution, but the 
quality of the paper is hampered by its experimental design. The paper felt generally unpolished, containing
frequent grammatical errors, imprecise language, and uncited statements.
My main issue with the paper is the experimental design. I am not convinced that we can 
draw valid conclusions from the experimental results for the following reasons:
  - The experiments are lacking important details. How many independent runs of the experiment 
    were the experimental results averaged over? All of the experiments have random initial conditions 
    (e.g. initialization of the network), and should be ran multiple times, not just once. 
    There's no error bars in any of the plots, so it's unclear whether AdaX really 
    does provide a statistically significant improvement over the baselines. 
    Similarly, the data in all the tables is quite similar, so without indicating the 
    spread of these estimates its impossible to tell whether these results are significant or not.
  - How were the hyperparameters and step-size schedules chosen? The performance of Adam, AMSGrad, and 
    RMSProp are quite sensitive to their hyperparameters, and the optimal hyperparameters are problem-dependent. 
    Some of the experiments just use the default hyperparameters; this is insufficient when trying to directly 
    compare the performance of these methods, as their performance can vary greatly with different values of
    these parameters. I'm not convinced that we should be drawing conclusions about the relative 
    performance of these algorithms from any of the experiments for this reason.
Of course, meaningful empirical results are not necessarily characterized by statistically outperforming the baselines. 
Well designed experiments can highlight important ways in which the performances differ, providing the community 
with a deeper understanding of the methods investigated. I would argue that the experiments in the paper do not 
achieve this either; the experiments do not provide any new intuition or understanding of the methods, showing 
only the relative performances in terms of learning curves on a somewhat random collection of supervised learning problems. Why were these specific problems chosen? What makes these problems ideal for showcasing the performance of AdaX? If AdaX is an improvement over Adam, why? What exactly is happening with it's effective step-sizes that leads 
to the better performance? Can you show how their step-sizes differ over time? 
Statements that need citation or revision:
  - "Adaptive optimization algorithms such as RMSProp and Adam... as well as weak performance 
     compared to the first order gradient methods such as SGD" (Abstract). This needs a citation. 
     Similarly, "AdaX outperforms various tasks of computer vision and natural language processing and can catch 
     up with SGD"; as above, I'm unaware of work (other than theoretical) that shows that SGD significantly 
     outperforms Adam in deep neural networks.
  -  "In the era of deep learning, SGD ... remains the most effective algorithm in training deep neural 
      networks" (Introduction). What are you referring to here? Vanilla SGD? Or are you including Adam etc here? 
      As above, this should have a citation. Adam's popularity is largely due to its effectiveness in training 
      deep neural networks.
  - "However, Adam has worse performance (i.e. generalization ability in testing stage) compared with SGD" 
     (Introduction). Citation needed.
  - In the last paragraph of the Introduction, you introduced AdaX twice: "To address the above issues, we propose a 
    new adaptive optimization method, termed AdaX, which guarantees convergence...", and, "To address 
    the above problems, we introduce a novel AdaX algorithm and theoreetically prove that it converges..."