Momentum is a simple and popular technique in deep learning for gradient-based optimizers. We propose a decaying momentum (DEMON) rule, motivated by decaying the total contribution of a gradient to all future updates. Applying DEMON to Adam leads to significantly improved training, notably competitive to momentum SGD with learning rate decay, even in settings in which adaptive methods are typically non-competitive. Similarly, applying DEMON to momentum SGD rivals momentum SGD with learning rate decay, and in some cases leads to improved performance. DEMON is trivial to implement and incurs limited extra computational overhead, compared to the vanilla counterparts.
This paper proposed a new decaying momentum rule to further improve neural network training. The idea is motivated by decaying the total contribution of a gradient to all future updates. The authors also extend this idea on to Adam and show that it improved upon the vanilla Adam.
- The intuition of using momentum decaying scheme is not quite clear to me. Why having a linear momentum decay schedule is better? This question is not answered in the paper. I hope the authors could provide more illustrative explanations regarding this.
- This paper focused essentially on empirical evaluations. I do appreciate that the authors conduction various experiments using different dataset and architecture but I do not understand why the authors want to separate the experiments into parts: adaptive methods and adaptive momentum methods. Not to mention the confusing names, it makes no sense to say the same words again for different optimizers. I would suggest the authors to combine the results together for better comparison.
- CM formulation mentioned in the paper is actually different from the commonly refer SGD with momentum implementation, which is the optimizer that is widely used in the community. Therefore, it is important to at least add SGD with momentum as one baseline in the experiments. 
- According to https://github.com/kuangliu/pytorch-cifar, Resnet18 on CIFAR10 can achieve at least 93% accuracy using simple SGD with momentum. It seems that the current reported results are not fully optimized. I would suggest the authors to check the parameter settings and make sure all the hyperparameters for baseline methods are fully tuned.
- Aside from SGD with momentum, the experiments part also lacks several important baselines. The author should also consider comparing the AdamW, Padam mentioned in the paper.