NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES
Principal Filter Analysis (PFA) is an easy to implement, yet effective method for neural network compression. PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprint. We propose two compression algorithms: the first allows a user to specify the proportion of the original spectral energy that should be preserved in each layer after compression, while the second is a heuristic that leads to a parameter-free approach that automatically selects the compression used at each layer. Both algorithms are evaluated against several architectures and datasets, and we show considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively. In our tests we also demonstrate that networks compressed with PFA achieve an accuracy that is very close to the empirical upper bound for a given compression ratio. Finally, we show how PFA is an effective tool for simultaneous compression and domain adaptation.
 The paper proposes to prune convolutional networks by analyzing the observed correlation between the filters of a same layer as expressed by the eigenvalue spectrum of their covariance matrix. The authors propose two strategies to decide of a compression level, one based on an eigenvalue threshold, the other one based on a heuristic based on the KL divergence between the observed eigenvalue distribution and the uniform one. This is a bit bizarre but does not require searching a parameter. Once one has decided the number of filters to keep, one can either retrain the network from scratch, or iteratively remove the most correlated filters, which, unsurprisingly, work better. The authors perform credible experiments on CIFAR-10 and CIFAR-100 that show the results one would expect. They should probably have run ImageNet experiments because many earlier papers on this topic use it as a benchmark and because the ImageNet size often reveals different behaviors. In conclusion, this is a very decent paper, but not a very exciting one. ------------- After reading the authors* response and their additional experiments, I still see this work as a very decent paper, but not a very exciting one. This is why I rank this paper somewhat above the acceptance threshold. I could be proven wrong if this approach becomes the method of choice to prune networks, but I would need to see a lot more comparisons to be convinced.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES
Principal Filter Analysis (PFA) is an easy to implement, yet effective method for neural network compression. PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprint. We propose two compression algorithms: the first allows a user to specify the proportion of the original spectral energy that should be preserved in each layer after compression, while the second is a heuristic that leads to a parameter-free approach that automatically selects the compression used at each layer. Both algorithms are evaluated against several architectures and datasets, and we show considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively. In our tests we also demonstrate that networks compressed with PFA achieve an accuracy that is very close to the empirical upper bound for a given compression ratio. Finally, we show how PFA is an effective tool for simultaneous compression and domain adaptation.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper proposes to prune convolutional networks by analyzing the observed correlation between the filters of a same layer as expressed by the eigenvalue spectrum of their covariance matrix. The authors propose two strategies to decide of a compression level, one based on an eigenvalue threshold, the other one based on a heuristic based on the KL divergence between the observed eigenvalue distribution and the uniform one. This is a bit bizarre but does not require searching a parameter. Once one has decided the number of filters to keep, one can either retrain the network from scratch, or iteratively remove the most correlated filters, which, unsurprisingly, work better. The authors perform credible experiments on CIFAR-10 and CIFAR-100 that show the results one would expect. They should probably have run ImageNet experiments because many earlier papers on this topic use it as a benchmark and because the ImageNet size often reveals different behaviors. In conclusion, this is a very decent paper, but not a very exciting one. ------------- After reading the authors* response and their additional experiments, I still see this work as a very decent paper, but not a very exciting one. This is why I rank this paper somewhat above the acceptance threshold. I could be proven wrong if this approach becomes the method of choice to prune networks, but I would need to see a lot more comparisons to be convinced.
assistant
 The paper proposes to prune convolutional networks by analyzing the observed correlation between the filters of a same layer as expressed by the eigenvalue spectrum of their covariance matrix. The authors propose two strategies to decide of a compression level, one based on an eigenvalue threshold, the other one based on a heuristic based on the KL divergence between the observed eigenvalue distribution and the uniform one. This is a bit bizarre but does not require searching a parameter. Once one has decided the number of filters to keep, one can either retrain the network from scratch, or iteratively remove the most correlated filters, which, unsurprisingly, work better. The authors perform credible experiments on CIFAR-10 and CIFAR-100 that show the results one would expect. They should probably have run ImageNet experiments because many earlier papers on this topic use it as a benchmark and because the ImageNet size often reveals different behaviors. In conclusion, this is a very decent paper, but not a very exciting one. ------------- After reading the authors* response and their additional experiments, I still see this work as a very decent paper, but not a very exciting one. This is why I rank this paper somewhat above the acceptance threshold. I could be proven wrong if this approach becomes the method of choice to prune networks, but I would need to see a lot more comparisons to be convinced.