Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference. We propose a novel model compression method that generates a sparse trained model without additional overhead: by allowing (i) dynamic allocation of the sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance). We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models. Moreover, their performance surpasses that of models generated by all previously proposed pruning schemes.
This work proposes a simple pruning method that dynamically sparsifies the network during training. This is achieved by performing at fixed intervals magnitude based pruning for either individual weights or entire neurons. While similar methods have been explored before, this work proposes a slight twist; instead of updating the weights of the model by following the gradient of the parameters of the dense model, they update the parameters of the dense model according to the gradients of the sparse model. Essentially, this corresponds to a variant of the straight-through estimator [1], where in the forward pass we evaluate the compressed model, but in the backward pass we update the model as if the compression didn’t take place. The authors argue that this process allows for ``feedback” in the pruning mechanism, as the pruned weights still receive gradient updates hence they can be ``re-activated” at later stages of training. They then provide a convergence analysis about the optimization procedure with such a gradient, and show that for strongly convex functions the method converges in the vicinity of the global optimum, whereas for non-convex functions it converges to the neighbourhood of a stationary point. Finally, the authors perform extensive experimental evaluation and show that their method is better than the baselines that they considered.
This work is in general well written and conveys the main idea in an effective manner.  It is also a timely contribution as sparse models / compression are important topics for the deep learning community. The overall method seems simple to implement, doesn’t introduce too many hyper-parameters and seem to work very well. For this reason I tend towards recommending for acceptance, provided that the authors address /comment on a couple of issues I found in the draft. 
More specifically:
- The connection to the error feedback is kind of loose and not well explained. After skimming Karimireddy et al. I noticed that 1. it evaluates the gradient at a point (i.e. the current estimate of the parameters), 2. compresses said gradient, 3. updates the parameters while maintaining the difference of the original w.r.t. the compressed gradient. In this sense, it seems a bit different that DPF, as your notation at the first equation of page 4 implies that you take the gradient of a different point, i.e. w_t + e_t instead of w_t. I believe that expanding a bit more about the connection would help in making the manuscript more clear.
- There seems to be a typo / error on your definition of an m-strongly convex function at the “convergence of Convex functions” paragraph. I believe it should be <\nabla f(v), w-v> <= f(w) - f(v) - 0.5 m ||w - v||^2, instead of <\nabla f(w), w-v> <= f(w) - f(v) - 0.5 m ||w - v||^2.
- The proposed gradient estimator seems to be an instance of the STE [1] estimator, that, as the authors mention, has been using at the Binary Connect algorithm. It would be interesting to see some more discussion about this similarity perhaps also expanding upon recent work that discusses the STE gradient as a form of coarse gradient [2].
- At section 5.2 the authors mention that “dynamic pruning methods, and in particular DPF, work on a different paradigm, and can still heavily benefit from fine-tuning”. This claim seems to contradict the results at Figure 4; there it seems that the masks have “converged” in the later stages of training, hence one could argue that the fine-tuning already happens thus it wouldn’t benefit DPF. I believe it would be interesting if the authors provide a similar plot as the one in Figure 4 but rather for the ResNet-20 network on CIFAR 10 (which seems to benefit heavily from FT). Do the masks still settle at the end of training (as it was the case for WideResNet-28-2) and if they do, why is fine-tuning still increasing the accuracy?
- Minor: Try to use consistent coloring at Figure 6 as while (a), (b) share the same color-coding, (c) is using a different one hence could be confusing.
[1] Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, Yoshua Bengio, Nicholas Léonard, Aaron Courville, 2013
[2] Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets, Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, Jack Xin, 2019