Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds
We present an efficient coresets-based neural network compression algorithm that sparsifies the parameters of a trained fully-connected neural network in a manner that provably approximates the network*s output. Our approach is based on an importance sampling scheme that judiciously defines a sampling distribution over the neural network parameters, and as a result, retains parameters of high importance while discarding redundant ones. We leverage a novel, empirical notion of sensitivity and extend traditional coreset constructions to the application of compressing parameters. Our theoretical analysis establishes guarantees on the size and accuracy of the resulting compressed network and gives rise to generalization bounds that may provide new insights into the generalization properties of neural networks. We demonstrate the practical effectiveness of our algorithm on a variety of neural network configurations and real-world data sets.
 The authors propose to reduce the size of fully connected neural networks, defined as the total number of nonzeros in the weight matrices, by calculating sensitivity scores for each incoming connection to a neuron, and randomly keeping only some of the incoming connections with probability proportional to their share of the total sensitivity. They provide a specific definition for the sensitivity scores and establish that the sparsified neural network, with constant probability for any sample from the training population, provides an output that is a small multiplicative factor away from the output of the unsparisfied neural network. The cost of the sparsification is essentially the application of the trained neural network to a small number of data points in order to compute the sensitivity scores Pros: - the method works empirically, in that their empirical evaluations on MNIST, CIFAR, and FashionMNIST classification problems show that the drop in accuracy is lower when the neural net is sparsified using their CoreNet algorithm and variations than when it is randomly sparsified or the neural network size is reduced by using SVD. - theory is provided to argue the consistency of the sparsified neural network Cons: - no comparison is made to the baseline of using matrix sparsification algorithms on the weight matrices themselves. I do not see why CoreNet should be expected to perform empirically better than simply using e.g. the entry-wise sampling scheme from "Near-optimal entrywise sampling for data matrices" by Achlioptas and co-authors, or earlier works addressing the same problem of sparsifying matrices. - the theory makes very strong assumptions (Assumptions 1 and 2) that are not explained or justified well. Both depend on the specific weight matrices being sparsified, and it isn*t clear a priori when the weight matrices obtained from whatever optimization procedure was used to train the neural net will be such that these assumptions hold. - despite the suggestions of the theory, the accuracy drop can be quite large in practice, as in the CIFAR panel of Figure 1 I think the ICLR audience will appreciate the attempt to provide a principled approach to decreasing the size of neural networks, but I do not think this approach is widely compelling as : (1) no true guaranteed control on the trade-off between accuracy loss and network size is available (2) empirically the method does not perform well consistently (3) comparisons with reasonable and informative baselines are missing Updated in response to author response: the inclusion of experimental comparisons with linear algebraic sparsification baselines, showing that the proposed method can be significantly more accurate, strengthens the appeal of the method.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds
We present an efficient coresets-based neural network compression algorithm that sparsifies the parameters of a trained fully-connected neural network in a manner that provably approximates the network*s output. Our approach is based on an importance sampling scheme that judiciously defines a sampling distribution over the neural network parameters, and as a result, retains parameters of high importance while discarding redundant ones. We leverage a novel, empirical notion of sensitivity and extend traditional coreset constructions to the application of compressing parameters. Our theoretical analysis establishes guarantees on the size and accuracy of the resulting compressed network and gives rise to generalization bounds that may provide new insights into the generalization properties of neural networks. We demonstrate the practical effectiveness of our algorithm on a variety of neural network configurations and real-world data sets.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors propose to reduce the size of fully connected neural networks, defined as the total number of nonzeros in the weight matrices, by calculating sensitivity scores for each incoming connection to a neuron, and randomly keeping only some of the incoming connections with probability proportional to their share of the total sensitivity. They provide a specific definition for the sensitivity scores and establish that the sparsified neural network, with constant probability for any sample from the training population, provides an output that is a small multiplicative factor away from the output of the unsparisfied neural network. The cost of the sparsification is essentially the application of the trained neural network to a small number of data points in order to compute the sensitivity scores Pros: - the method works empirically, in that their empirical evaluations on MNIST, CIFAR, and FashionMNIST classification problems show that the drop in accuracy is lower when the neural net is sparsified using their CoreNet algorithm and variations than when it is randomly sparsified or the neural network size is reduced by using SVD. - theory is provided to argue the consistency of the sparsified neural network Cons: - no comparison is made to the baseline of using matrix sparsification algorithms on the weight matrices themselves. I do not see why CoreNet should be expected to perform empirically better than simply using e.g. the entry-wise sampling scheme from "Near-optimal entrywise sampling for data matrices" by Achlioptas and co-authors, or earlier works addressing the same problem of sparsifying matrices. - the theory makes very strong assumptions (Assumptions 1 and 2) that are not explained or justified well. Both depend on the specific weight matrices being sparsified, and it isn*t clear a priori when the weight matrices obtained from whatever optimization procedure was used to train the neural net will be such that these assumptions hold. - despite the suggestions of the theory, the accuracy drop can be quite large in practice, as in the CIFAR panel of Figure 1 I think the ICLR audience will appreciate the attempt to provide a principled approach to decreasing the size of neural networks, but I do not think this approach is widely compelling as : (1) no true guaranteed control on the trade-off between accuracy loss and network size is available (2) empirically the method does not perform well consistently (3) comparisons with reasonable and informative baselines are missing Updated in response to author response: the inclusion of experimental comparisons with linear algebraic sparsification baselines, showing that the proposed method can be significantly more accurate, strengthens the appeal of the method.
assistant
 The authors propose to reduce the size of fully connected neural networks, defined as the total number of nonzeros in the weight matrices, by calculating sensitivity scores for each incoming connection to a neuron, and randomly keeping only some of the incoming connections with probability proportional to their share of the total sensitivity. They provide a specific definition for the sensitivity scores and establish that the sparsified neural network, with constant probability for any sample from the training population, provides an output that is a small multiplicative factor away from the output of the unsparisfied neural network. The cost of the sparsification is essentially the application of the trained neural network to a small number of data points in order to compute the sensitivity scores Pros: - the method works empirically, in that their empirical evaluations on MNIST, CIFAR, and FashionMNIST classification problems show that the drop in accuracy is lower when the neural net is sparsified using their CoreNet algorithm and variations than when it is randomly sparsified or the neural network size is reduced by using SVD. - theory is provided to argue the consistency of the sparsified neural network Cons: - no comparison is made to the baseline of using matrix sparsification algorithms on the weight matrices themselves. I do not see why CoreNet should be expected to perform empirically better than simply using e.g. the entry-wise sampling scheme from "Near-optimal entrywise sampling for data matrices" by Achlioptas and co-authors, or earlier works addressing the same problem of sparsifying matrices. - the theory makes very strong assumptions (Assumptions 1 and 2) that are not explained or justified well. Both depend on the specific weight matrices being sparsified, and it isn*t clear a priori when the weight matrices obtained from whatever optimization procedure was used to train the neural net will be such that these assumptions hold. - despite the suggestions of the theory, the accuracy drop can be quite large in practice, as in the CIFAR panel of Figure 1 I think the ICLR audience will appreciate the attempt to provide a principled approach to decreasing the size of neural networks, but I do not think this approach is widely compelling as : (1) no true guaranteed control on the trade-off between accuracy loss and network size is available (2) empirically the method does not perform well consistently (3) comparisons with reasonable and informative baselines are missing Updated in response to author response: the inclusion of experimental comparisons with linear algebraic sparsification baselines, showing that the proposed method can be significantly more accurate, strengthens the appeal of the method.