Deep neural networks (DNNs), although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work has been conducted on DNN model compression or pruning. However, most of the previous work has taken heuristic approaches. This work proposes a progressive weight pruning approach based on ADMM (Alternating Direction Method of Multipliers), a powerful technique to deal with non-convex optimization problems with potentially combinatorial constraints. Motivated by dynamic programming, the proposed method reaches extremely high pruning rate by using partial prunings with moderate pruning rates. Therefore, it resolves the accuracy degradation and long convergence time problems when pursuing extremely high pruning ratios. It achieves up to 34× pruning rate for ImageNet data set and 167× pruning rate for MNIST data set, significantly higher than those reached by existing work in the literature. Under the same number of epochs, the proposed method also achieves better convergence and higher compression rates. The codes and pruned DNN models are avilable in the link: bit.ly/2zxdlss.
This paper focus on weight pruning for neural network compression. The proposed method is based on ADMM optimization method for neural network loss with constraint on the l_0 norm of weights, proposed in Zhang et al. 2018b. Two improvements, masked retraining and progressive pruning, are introduced. Masked retraining set the weights to zero at early stages and stop updating those weights. Progressing pruning keeps a buffer of partial pruning results and select the best performed model for further pruning. The proposed method achieves 30x compression rate for AlexNet and VGG for ImageNet.
I have the following concerns about the proposed method. 
- It is unclear to me what is the benefit of ADMM for solving the sparse regularized NN optimization problem. Why is it better than projected gradient descent or proximal gradient method used in previous network pruning? I understand the proposed method is based on Zhang et al. 2018b, but a strong argument will support the draft.
- I fail to understand the claim ``at convergence, the pruned DNN model will not be exactly sparse’’ in section 2.3. Z will always be sparse after the projection step in (5). At convergence, the linear constraint should be satisfied, which makes W = Z to be sparse. 
- Please describe the the proposed method in detail. The current description is very vague and I do not think it can be reimplemented based on the current draft. In each outer loop of ADMM, (4)(5) and dual update is applied (I consider solving (4) is the inner loop). How is the mask generated and fit into these equations? For progressive pruning, it looks to me there is an outer loop outside the outer loop of ADMM. Please provide details on how many iterations, and how the compression rate is decided for each iteration.
- The hyper-parameters of the proposed method is unclear. It is a bit strange the optimization parameter \rho could control the pruning rate (section 3.1). As described before, I guess the proposed method has three loops. How is the iterations counted, like for Figure 3. Please clarify the experiments are fair comparison, the better results are not because of more weight updates from the three loops. 
- It is unclear what is the benefit of masked retraining. It looks to me this kind of greedy approach will harm the performance (I have to guess if a weight is masked to be zero, it will never be updated or recovered). What happens if there are a lot of weights (Z in (5)) are zero at the early stage?
- The progressive pruning looks heuristic and I am not convinced the buffer is necessary. There is always a progressive pruning trace that can directly lead to the results without selecting from candidates. For example, in Figure 2, we can just train model from 15x to 24x to 30x.  
- The following works are related. 
Li et al. Pruning Filters for Efficient ConvNets. ICLR 2017
Alvarez et al. Learning the number of neurons in deep networks. NIPS 2016
=============== after rebuttal ===================
I appreciate the authors' feedback and slightly raise the score. 
Though the compression results look good, I still have some concerns about the method. The motivation of the proposed method is not strong. The proposed mask is greedy and sounds ad-hoc. The proposed progressive pruning looks expensive. 
The proposed method looks time consuming. For the experiments, I would love to see the training time comparing with baselines in table 1, not only the ADMM method in table 2. A fair comparison could be wall-clock time, or number of gradient updates for neural networks.