Recently, there has been growing interest in methods that perform neural network compression, namely techniques that attempt to substantially reduce the size of a neural network without performance degradation. In this paper, we propose a simple compression algorithm that incorporates pruning within scalar quantization, whose non-probabilistic nature results in minimal change and overhead to standard network training. The key idea in our approach is to modify the original optimization problem by adding K independent Gaussian priors and a sparsityinducing prior over the parameters. We show that our approach is easy to implement using existing neural network libraries, generalizes 1 and 2 regularization, and elegantly enforces parameter tying/pruning constraints on all parameters across the network. Experimentally, we demonstrate that our method yields state-of-the-art compression on several standard benchmarks with minimal loss in accuracy while requiring significantly less hyperparameter tuning compared with related, competing approaches.
As the authors mentioned, weight-sharing and pruning are not new to neural network compression. The proposed method resembles a lot with the deep compression work (Han et. al. 2016), with the distinction of clustering across different layers and a Lasso regularizer to encourage sparsity of the weights. Even though the change seems minimal, the authors has demonstrated the effectiveness on the benchmark.
But the description of the optimization strategy in Section 3 needs some refinement. In the soft-tying stage, why only the regularizer (1) is considered, not the sparsity one? In the hard-tying stage, would the clustering change in each iteration? If not, this has reduced to the constrained problem as in the Hashed Compression work (Chen et. al. 2015) where the regularizer (1) has no effect since the clustering is fixed and all the weights in the same cluster are equal. Even though it is claimed that the proposed method does not require a pre-trained model to initialize, the soft-tying stage seems to take the responsibility to "pre-train" the model.
The experiment section is a weak point. It is much less convincing with no comparison result of compression on large neural networks and large datasets. The only compression result on large neural network (VGG-11) comes with no baseline comparisons. But it already tells something: 1) what is the classification result for reference network without compression? 2) the compression ratio has significantly reduced comparing with those for MNIST. It is hard to say if the compression performance could generalize to large networks.
Also, it would be good to have an ablation test on different parts of the objective function and the two optimization stages to show the importance of each part, especially the removal of the soft-tying stage and the L1 regularizer versus a simple pruning technique after each iteration. This maybe a minor issue, but would be interesting to know: what would the compression performance be if the classification accuracy maintains the same level as that of the deep compression. As discussed in the paper, it is a trade-off between accuracy and compression. The network could be compressed to very small size but with significant accuracy loss.
Some minor issues:
- In Section 1, the authors discussed a bunch of pitfalls of existing compression techniques, such as large number of parameters, local minimum issues and layer-wise approaches. It would be clearer if the authors could explicitly and succinctly discuss which pitfalls are resolved and how by the proposed method towards the end of the Introduction section. 
- In Section 4.2, the authors discussed the insensitivity of the proposed method to switching frequency. But there is no quantitative results shown to support the claims.
- What is the threshold for pruning zero weight used in Table 2?
- There are many references and comparisons missing: Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations in NIPS 17 for instance. This paper also considers quantization for compression which is related to this work.