There exist several stochastic optimization algorithms. However in most cases, it is difficult to tell for a particular problem which will be the best optimizer to choose as each of them are good. Thus, we present a simple and intuitive technique, when applied to first order optimization algorithms, is able to improve the speed of convergence and reaches a better minimum for the loss function compared to the original algorithms. The proposed solution modifies the update rule, based on the variation of the direction of the gradient during training. We conducted several tests with Adam and AMSGrad on two different datasets. The preliminary results show that the proposed technique improves the performance of existing optimization algorithms and works well in practice.
The paper considers a simplistic extension of first order methods typically used for neural network training. Apart from the basic idea the paper's actual algorithm is hard to read because it is full of lacking definitions. I have tried to piece together whatever I could by reading the proof. The algorithm box is very unclear. For instance the * operator is undefined. 
To the best of my understanding which the paper changes the update by first checking whether the gradient has the same direction as the previous gradient if yes it uses the component wise maximum of the new gradient and the previous gradient in the update and otherwise it uses the new gradient. Now whether this if condition is checked component wise or an angle between the two vectors is completely unclear. 
I will really suggest the authors to at least write their algorithm with clarity. Further while stating the theorem there are undefined parameter and even the objective Regret has not been defined anywhere. Further the theorem which I could not verify due to similar unclarity shows I believe the same convergence result as AMSGrad and hence there is no theoretical advantage for the proposed algorithm. In terms of practice further I do not see a significant advantage and it could result be a step size issue . The authors do not say that they do a search over the hyper parameters. 
On a philosophical level it is unclear what the motivation behind this particular change to any algorithm is. It would be good to discuss what additional advantage is added on top of acceleration. Note that the method feels very much like acceleration.