Sparse neural networks have been shown to be more parameter and compute efficient compared to dense networks and in some cases they are even successfully used to decrease wall clock inference times. There is a large body of work on training dense networks to yield sparse networks for inference (Molchanov et al., 2017;Zhu & Gupta, 2018;Louizos et al., 2017;Guo et al., 2016). This limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-to-sparse training methods. Our method updates the topology of the network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results with ResNet-50, MobileNet v1 and MobileNet v2 on the ImageNet-2012 dataset, WideResNets on the CIFAR-10 dataset and RNNs on the WikiText-103 dataset. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static.
This paper proposes a pruning technique called RigL, which performs sparse initialization of the network weights, and allows the network to grow weights during training. The sparse initialization makes the pruning algorithm to be memory- and computation- efficient, unlike existing models that starts with dense network weights. The authors train RigL with three different sparsity distributions that considers the number of input and output nodes, and validate them on various deep convnets for training on ImageNet, on which the model outperforms several existing sparsification methods and even dense counterparts. 
Pros
- The proposed model, RigL, is memory- and computation- efficient, and thus allows to train a large network in an efficient manner.
 
- RigL obtains impressive sparsification performance, even yielding sparse networks that outperform their dense counterparts.
Cons
- The idea of starting from a small, sparse network and expanding it is not novel. DEN [Yoon et al. 18] proposed the same bottom-up approach with sparsely initialized networks, while they allowed to increase the number of neurons at each layer and focused more on continual learning. The authors should compare the two methods both conceptually and experimentally. 
- The method is more like a set of heuristics rather than a principled approach, which makes it less appealing. This is not really an issue if the paper includes extensive experimental validation and in-depth analysis, but this is not the case. 
 
- The experimental validation is largely lacking, as the authors only perform experiments on ImageNet and do not compare against recent state-of-the-art Bayesian sparsification methods (SBP, VIB, L0-regularization). Without such extensive experimental validation, it is uncertain whether the result will generalize, given the highly empirical nature of the work. 
In sum, although I believe that the paper proposes a very practical method that is easy to implement and is promising, due to lack of experimental validation against a similar approach, state-of-the-art sparsification methods, and results on more datasets, I temporarily provide the rating of weak reject. I may change my opinion if the authors provide those results during the rebuttal period.
[Yoon et al. 18] Lifelong learning with dynamically expandable networks, ICLR 2018