In spite of the slow convergence, stochastic gradient descent (SGD) is still the most widely used optimization method due to its outstanding generalization ability and simplicity. On the other hand, adaptive methods have attracted much more attention of optimization and machine learning communities, both for the leverage of life-long information and for the deep and fundamental mathematical theory. Taking the best of both worlds is the most exciting and challenging question in the field of optimization for machine learning. In this paper, we revisit existing adaptive gradient methods from a novel point of view, which reveals a fresh understanding of momentum. Our new intuition empowers us to remove the second moment in Adam without the loss of performance. Based on our view, we propose a novel optimization method, the acute adaptive momentum (Acutum). To the best of our knowledge, Acutum is the first adaptive gradient method without second moments. Experimentally, we demonstrate that our method has a faster convergence rate than Adam/Amsgrad, and generalizes as well as SGD with momentum. We also provide a convergence analysis of our proposed method to complement our intuition.
This paper attempts to remove the use of the second moment in Adam in order to improve the generalization ability of Adam. 
- Apriori, it is not clear why removing the second moment is important. Does it improve the generalization or decrease the runtime substantially?
- Please clarify how our method compares against Yogi (Adaptive Methods for Nonconvex Optimization, Zaheer' 2018) and the more recent RADAM (ON THE VARIANCE OF THE ADAPTIVE LEARNING
RATE AND BEYOND, Liu 2019) , both theoretically and empirically. 
- The paper is meandering and confusing. State your update/algorithm and then explain how it compares against the other methods. Besides, there might be technical problems with it. 
See the detailed review below:
- Section 1: "the generalization results (of adaptive methods) cannot be as good as SGD". Please cite the relevant papers, (for example, Wilson. 2017)  that show this empirically. 
-  Section 1: "the proposed algorithm outperforms Adam in convergence speed" - Please clarify what "convergence speed" refers to. Is it the number of gradient evaluations, the rate of convergence or the wall-clock time. Please state how did you conclude this. 
- Section 2: The update rule of Adagrad is incorrect. The step-size is constant \alpha and it is decreased over time because of the v_t term. 
- Section 3: There is no guarantee on the approximation in Equation 5. Young's inequality and the resulting upper bound can be quite loose. 
- Section 3: In general, it is not possible to have an update that decreases the loss on the current batch, but does not increase the previous batches loss. It is always possible to construct a counter-example to this. 
- "In practice, the computational cost of computing" the gradient for all i is expensive. Indeed, this is batch gradient descent. I am not sure how this is relevant to the discussion in the paper. 
- Cite and compare against the variance reduced methods (Stochastic Average Gradient, Schmidt, 2013; SVRG, Johnson, 2013) as these try to "approximate" the full gradient in order to decrease the variance. 
- The derivation/formulation of Equation 8 is not clear to me. Why is the \hat{m} normalized? 
- In algorithm 1, it seems you need to choose the sequence of step-sizes \alpha_t and \beta_t. How is this an adaptive method then? How are these sequences chosen theoretically and practically? Please clarify this. 
- Section 4: Please compare the resulting regret bound to that of Adam, Adagrad and AMSgrad. Why does \alpha_t = O(1/t)? If we have to decrease the step-size according to a sequence, why should I not use standard SGD?
- Section 5: "we decay the learning rate by 0.1 every 50 epochs" This is not aligned with either the theory or the algorithm you proposed.