DATA-DEPENDENT CORESETS FOR COMPRESSING NEURAL NETWORKS WITH APPLICATIONS TO GENER- ALIZATION BOUNDS
We present an efficient coresets-based neural network compression algorithm that sparsifies the parameters of a trained fully-connected neural network in a manner that provably approximates the network's output. Our approach is based on an importance sampling scheme that judiciously defines a sampling distribution over the neural network parameters, and as a result, retains parameters of high importance while discarding redundant ones. We leverage a novel, empirical notion of sensitivity and extend traditional coreset constructions to the application of compressing parameters. Our theoretical analysis establishes guarantees on the size and accuracy of the resulting compressed network and gives rise to generalization bounds that may provide new insights into the generalization properties of neural networks. We demonstrate the practical effectiveness of our algorithm on a variety of neural network configurations and real-world data sets.
4. Analytical results establishing guarantees on the approximation accuracy, size, and generalization of the compressed neural network.
5. Evaluations on real-world data sets that demonstrate the practical effectiveness of our algorithm in compressing neural network parameters and validate our theoretical results.
Given an additively decomposable function F(X, Q) = sum_over_x_in_X cost(x, Q), one can approximate it using either random sampling of x in X (unbiased, possibly high variance), or using importance sampling and replace the sum_over_x with a sum_over_coreset importance_of_a_point * cost(x, Q) which if properly defined can be both unbiased and have low variance [1]. In this work the authors consider the weighted sum of activations as F and suggest that for each neuron we can subsample the incoming edges. To construct the importance sampling strategy the authors adapt the classic notion of sensitivity from the coreset literature. Then, one has to carefully balance the approximation quality from one layer to the next and essentially union bound the results over all layers and all sampled points. The performed analysis is sound (up to my knowledge).
Pro:
- I commend the authors for a clean and polished writeup.
- The analysis seems to be sound (apart from the issues discussed below)
- The experimental results look promising, at least in the limited setup.
Con:
- There exists competing work with rigorous guarantees, for example [2].
- The analysis hinges on two assumptions which, in my opinion, make the problem feasible: having (sub) exponential tails allows for strong concentration results, and with proper analysis (as done by the authors), the fact that the additively decomposable function can be approximated given well-behaving summands is not surprising. The analysis is definitely non-trivial and I commend the authors for a clean writeup.
- While rigorous guarantees are lacking for some previous work, previously introduced techniques were shown to be extremely effective in practice and across a spectrum of tasks. As the guarantees arguably stem from the assumptions 1 and 2, I feel that it’s unfair to not compare to those results empirically. Hence, failing to compare to results of at least [2, 3] is a major drawback of this work.
- The result holds for n points drawn from P. However, in practice the network might receive essentially arbitrary input from P at inference time. Given that we need to decide on the number of edges to preserve apriori, what are the implications?
- The presented bounds should be discussed on an intuitive level (i.e. the number of non zero entries is approximately cubic in L).
I consider this to be a well-executed paper which brings together the main ideas from the coreset literature and shows one avenue of establishing provable results. However, given that no comparison to the state-of-the-art techniques is given I'm not confident that the community will apply these techniques in practice. On the other hand, the main strength -- the theoretical guarantees -- hinge on the introduced assumptions. As such, without additional empirical results demonstrating the utility with respect to the state-of-the-art methods (for the same capacity in terms of NNZ) I cannot recommend acceptance.
[1] https://arxiv.org/abs/1601.00617
[2] papers.nips.cc/paper/6910-net-trim-convex-pruning-of-deep-neural-networks-with-performance-guarantee
[3] https://arxiv.org/abs/1510.00149
========
Thank you for the detailed responses. Given the additional experimental results and connections to existing work, I have updated my score from 5 to 6.