We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to identify layers and weights which reduce the error efficiently. Sparse momentum redistributes pruned weights across layers according to the mean momentum magnitude of each layer. Within a layer, sparse momentum grows weights according to the momentum magnitude of zero-valued weights. We demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10, and ImageNet, decreasing the mean error by a relative 8%, 15%, and 6% compared to other sparse algorithms. Furthermore, we show that sparse momentum reliably reproduces dense performance levels while providing up to 5.61x faster training. In our analysis, ablations show that the benefits of momentum redistribution and growth increase with the depth and size of the network.
In this paper, the authors propose a sparse momentum algorithm for doing efficient sparse training. The technique relies on identifying weights in a layer that do not have an effect on the error, pruning them, and redistributing and growing them across layers. The technique is compared against other recent algorithms on a range of models.
The paper is well written, and very easy to read. The proposed algorithm looks interesting and seems to empirically work well. 
I am, however, a bit confused with how the sparse momentum algorithm is discussed in this paper. In the paper, momentum is intuitively presented as an algorithm that reduces the variance of the noise in the gradients. However, there are a number of papers that show that this is not the case (and if anything it is the opposite). Further, a few recent papers show that momentum works better than SGD only for learning rates that are not too small, and this is because it averages out the gradients in the high-curvature directions (these are the directions where the gradients switch signs) and makes them stable in these directions, thus allowing larger steps in the low curvature directions. See for example the following two papers:
Momentum Enables Large Batch Training. Samuel L Smith, Erich Elsen, Soham De. ICML Workshop on Physics for Deep Learning, 2019.
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model. Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E Dahl, Christopher J Shallue, Roger Grosse. NeurIPS 2019.
Given these papers, can the authors comment on what they think is the reason for the effectiveness of their sparse momentum algorithm? These papers seem to indicate to me that an interesting (and important) ablation study would be to compare using just the gradients vs using momentum for the sparse training algorithm for both a small batch and a large batch. Without this ablation study, it is a bit unclear to me why/when this algorithm is working well, and this primarily explains my current score.
The experimental results look impressive, although I am not very aware of other work in sparse training, so unfortunately it is harder for me to properly evaluate the significance of the empirical results in this paper, and whether the numbers reported are indeed a significant improvement over current state-of-the-art algorithms. I do have a few questions about the design and the results of the experiments presented:
1. Because the momentum of zero-valued weights are used, does that mean that the gradient over all weights are required to be taken at each step? How much does this affect training time in your experiments?
2. How were the learning rates decided, and why are they kept fixed across methods? It seems feasible that the optimal learning rate could vary highly between methods?
==========================================
Edit after rebuttal:
I thank the authors for the very detailed response and for doing the additional experiments requested. Due to the thoroughness of the response, I am increasing my score to a weak accept. I do however have a couple of comments regarding the author response:
1. I do not agree that keeping the learning rate fixed across methods is the right approach. Many different things might interact with each other to change the optimal learning rate, and it is always more interesting to see the optimal performance attained by a method through a grid search. As long as the tuning budget across all methods are somewhat similar, I think the experiments would be more informative when doing a learning rate sweep. Further, while doing the sweep, the authors should be careful that the optimal learning rate does not lie on the edge of the grid search. This is what happens in the experiments reported in the authors' response titled "Results from learning rate grid search" where the optimal learning rate for all models lie at one of the extremes of the grid search (0.09).
2. I would request the authors to slightly rewrite certain parts of their paper so as not to imply that momentum decreases the variance of the gradients in general. There are many papers (including the ones I mentioned in my review) that specifically show momentum does not have this effect. All of these papers however consider a different setting from the one considered in this paper (where there is a parameter redistribution step), so it is not immediately clear whether momentum has the same effect in this case. However, unless there is very specific evidence to show a variance reduction effect, I think putting in that intuition might be misleading. 
It is interesting to see that momentum helps for small batches in this setting, and I agree that further investigation into this would be an interesting future direction. Note that there is a concurrent submission on sparse training (https://openreview.net/forum?id=ryg7vA4tPB&noteId=ryg7vA4tPB) that seems to show that higher momentum values (0.99) does better when considering large batches. So perhaps additional and more careful experiments need to be done about this before these additional experimental results can be put in the paper.