The growing interest to implement Deep Neural Networks (DNNs) on resourcebound hardware has motivated the innovation of compression algorithms. Using these algorithms, DNN model sizes can be substantially reduced, with little-to-no accuracy degradation. This is achieved by either eliminating components from the model, or penalizing complexity during training. While both approaches demonstrate considerable compressions, the former often ignores the loss function during compression while the latter produces unpredictable compressions. In this paper, we propose a technique that directly minimizes both the model complexity and the changes in the loss function. In this technique, we formulate compression as a constrained optimization problem, and then present a solution for it. We will show that using this technique, we can achieve competitive results.
1. Summary
This paper introduced a method to learn a compressed version of a neural network such that the loss of the compressed network doesn't dramatically change.
2. High level paper
- I believe the writing is a bit sloppy. For instance equation 3 takes the minimum over all m in C but C is defined to be a set of c_1, ..., c_k, and other examples (see section 4 below). This is unfortunate because I believe this method, which takes as input a large complex network and compresses it so the loss in accuracy is small, would be really appealing to companies who are resource constrained but want to use neural network models.
3. High level technical
- I'm confused at the first and second lines of equation (19). In the first line, shouldn't the first term not contain \Delta W ? In the second line, shouldn't the first term be \tilde{\mathcal{L}}(W_0 + \Delta W) ?
- For CIFAR-10 and SVHN you're using Binarized Neural Networks and the two nice things about this method are (a) that the memory usage of the network is very small, and (b) network operations can be specialized to be fast on binary data. My worry is if you're compressing these networks with your method are the weights not treated as binary anymore? Now I know in Binarized Neural Networks they keep a copy of real-valued weights so if you're just compressing these then maybe all is alright. But if you're compressing the weights _after_ binarization then this would be very inefficient because the weights won't likely be binary anymore and (a) and (b) above no longer apply.
- Your compression ratio is much higher for MNIST but your accuracy loss is somewhat dramatic, especially for MNIST (an increase of 0.53 in error nearly doubles your error and makes the network worse than many other competing methods: http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html#4d4e495354). What is your compression ratio for 0 accuracy loss? I think this is a key experiment that should be run as this result would be much easier to compare with the other methods.
- Previous compression work uses a lot of tricks to compress convolutional weights. Does your method work for convolutional layers?
- The first paper to propose weight sharing was not Han et al., 2015, it was actually:
Chen W., Wilson, J. T., Tyree, S., Weinberger K. Q., Chen, Y. "Compressing Neural Networks with the Hashing Trick" ICML 2015
Although they did not learn the weight sharing function, but use random hash functions.
4. Low level technical
- The end of Section 2 has an extra 'p' character
- Section 3.1: "Here, X and y define a set of samples and ideal output distributions we use for training" this sentence is a bit confusing. Here y isn't a distribution, but also samples drawn from some distribution. Actually I don't think it makes sense to talk about distributions at all in Section 3.
- Section 3.1: "W is the learnt model...\hat{W} is the final, trained model" This is unclear: W and \hat{W} seem to describe the same thing. I would just remove "is the learnt model and"
5. Review summary
While the trust-region-like optimization of the method is nice and I believe this method could be useful for practitioners, I found the paper somewhat confusing to read. This combined with some key experimental questions I have make me think this paper still needs work before being accepted to ICLR.