Data-Independent Neural Pruning via Coresets
Previous work showed empirically that large neural networks can be significantly reduced in size while preserving their accuracy. Model compression became a central research topic, as it is crucial for deployment of neural networks on devices with limited computational and memory resources. The majority of the compression methods are based on heuristics and offer no worst-case guarantees on the trade-off between the compression rate and the approximation error for an arbitrarily new sample. We propose the first efficient, data-independent neural pruning algorithm with a provable trade-off between its compression rate and the approximation error for any future test sample. Our method is based on the coreset framework, which finds a small weighted subset of points that provably approximates the original inputs. Specifically, we approximate the output of a layer of neurons by a coreset of neurons in the previous layer and discard the rest. We apply this framework in a layer-by-layer fashion from the top to the bottom. Unlike previous works, our coreset is data independent, meaning that it provably guarantees the accuracy of the function for any input , including an adversarial one. We demonstrate the effectiveness of our method on popular network architectures. In particular, our coresets yield 90% compression of the LeNet-300-100 architecture on MNIST while improving the accuracy.
The authors proposed a model compressing method based on coreset framework. The goal of the paper is to reduce the number of neurons. The basic idea is sampling the neurons on each layer with probability equal to the neuron*s max share among all the outputs to the next layer, and updating the weights associated with the remained neurons. Another main contribution is the authors provided theoretical analysis to guarantee the accuracy vs compression trade-off for all possible inputs. Pros: The proposed method is easy to understand and seems to make sense. The theoretical analysis seems strong. The experiment results on two datasets show the proposed method achieved high compression rate and improvement of accuracy. Cons: Despite the theoretical guarantee, it is not as clear on the value of the proposed method in real world. I would be better to test on more datasets and networks to verify the effectiveness of the proposed compressing method, as it claimed to be data-independent. Although the method achieved very good experiment results, its contribution to the high accuracy is unclear, since the networks were fine-tuned after the compression. So how do we exactly evaluate the accuracy vs compression trade-off when there is no such trade-off shown in the experiments? Questions and suggestions: In the Fig 2, it seems that the performances of the proposed method and the percentile-based method should be close to each other, and the uniform sampling method should be worse than them. However the results are opposite. If it was not incorrect labeling in the figure, it would be good to add some analyses about this result. To solve the second point in *Cons*, is it possible to show the accuracy of the compressed model without fine-tune? Or still fine-tune the model but simply set u(q) = w(q) in the line 8 algorithm 1?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Data-Independent Neural Pruning via Coresets
Previous work showed empirically that large neural networks can be significantly reduced in size while preserving their accuracy. Model compression became a central research topic, as it is crucial for deployment of neural networks on devices with limited computational and memory resources. The majority of the compression methods are based on heuristics and offer no worst-case guarantees on the trade-off between the compression rate and the approximation error for an arbitrarily new sample. We propose the first efficient, data-independent neural pruning algorithm with a provable trade-off between its compression rate and the approximation error for any future test sample. Our method is based on the coreset framework, which finds a small weighted subset of points that provably approximates the original inputs. Specifically, we approximate the output of a layer of neurons by a coreset of neurons in the previous layer and discard the rest. We apply this framework in a layer-by-layer fashion from the top to the bottom. Unlike previous works, our coreset is data independent, meaning that it provably guarantees the accuracy of the function for any input , including an adversarial one. We demonstrate the effectiveness of our method on popular network architectures. In particular, our coresets yield 90% compression of the LeNet-300-100 architecture on MNIST while improving the accuracy.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors proposed a model compressing method based on coreset framework. The goal of the paper is to reduce the number of neurons. The basic idea is sampling the neurons on each layer with probability equal to the neuron*s max share among all the outputs to the next layer, and updating the weights associated with the remained neurons. Another main contribution is the authors provided theoretical analysis to guarantee the accuracy vs compression trade-off for all possible inputs. Pros: The proposed method is easy to understand and seems to make sense. The theoretical analysis seems strong. The experiment results on two datasets show the proposed method achieved high compression rate and improvement of accuracy. Cons: Despite the theoretical guarantee, it is not as clear on the value of the proposed method in real world. I would be better to test on more datasets and networks to verify the effectiveness of the proposed compressing method, as it claimed to be data-independent. Although the method achieved very good experiment results, its contribution to the high accuracy is unclear, since the networks were fine-tuned after the compression. So how do we exactly evaluate the accuracy vs compression trade-off when there is no such trade-off shown in the experiments? Questions and suggestions: In the Fig 2, it seems that the performances of the proposed method and the percentile-based method should be close to each other, and the uniform sampling method should be worse than them. However the results are opposite. If it was not incorrect labeling in the figure, it would be good to add some analyses about this result. To solve the second point in *Cons*, is it possible to show the accuracy of the compressed model without fine-tune? Or still fine-tune the model but simply set u(q) = w(q) in the line 8 algorithm 1?
assistant
The authors proposed a model compressing method based on coreset framework. The goal of the paper is to reduce the number of neurons. The basic idea is sampling the neurons on each layer with probability equal to the neuron*s max share among all the outputs to the next layer, and updating the weights associated with the remained neurons. Another main contribution is the authors provided theoretical analysis to guarantee the accuracy vs compression trade-off for all possible inputs. Pros: The proposed method is easy to understand and seems to make sense. The theoretical analysis seems strong. The experiment results on two datasets show the proposed method achieved high compression rate and improvement of accuracy. Cons: Despite the theoretical guarantee, it is not as clear on the value of the proposed method in real world. I would be better to test on more datasets and networks to verify the effectiveness of the proposed compressing method, as it claimed to be data-independent. Although the method achieved very good experiment results, its contribution to the high accuracy is unclear, since the networks were fine-tuned after the compression. So how do we exactly evaluate the accuracy vs compression trade-off when there is no such trade-off shown in the experiments? Questions and suggestions: In the Fig 2, it seems that the performances of the proposed method and the percentile-based method should be close to each other, and the uniform sampling method should be worse than them. However the results are opposite. If it was not incorrect labeling in the figure, it would be good to add some analyses about this result. To solve the second point in *Cons*, is it possible to show the accuracy of the compressed model without fine-tune? Or still fine-tune the model but simply set u(q) = w(q) in the line 8 algorithm 1?