INTERPRETING CNN COMPRESSION USING INFORMATION BOTTLENECK
In this paper, we first investigate the representation learned in convolutional neural networks at the filter-wise granularity by computing the mutual information between channels of higher conv-layers and input or output variables. Then we identify the approximate minimal sufficient statistics of learned representation based on the information bottleneck principle and propose a novel approach to automatically compress a neural network. This approach prunes a large trained network structurally and automatically by extracting relevant information backpropagately layer by layer in the post-training phase. Our experimental results match the two fundamental data processing inequalities, and prove that mutual information is a fundamental element for examining the efficiency of the internal representations at the filter-wise granularity. In addition, using the information bottleneck principle to interpret structure compression is an efficient method to get closer to the information theoretic limit of compression/prediction problem. Finally, from the observed results, we argue that compression is causally linked to the improved generalization performance.
The paper proposes a novel way of compressing a CNN by selecting the subset of convolutional filters that maximises the IB criterion. Based on this idea the authors propose an automatic procedure to prune non-informative filters from a trained network and show empirically that, with minimal impact on performance. The idea of the paper is interesting but I feel the relative novelty of the approach is not sufficient to make it eligible as a full conference paper. The idea of feature selection using information gain is a classical approach in standard applied ML literature and the only difference I see in the current approach that it is applied to convolutional filters and not to features themselves. Furthermore, the writing of the paper needs to be revisited, there are many instances of sentences that are confusing or not fully finished such as: - So pruning these high informative filters affects to predict the objects correctly - […] in which network are trained to regularize most of neurons approximately uninformative. - […] then prune the high-informative filters incrementally in the precise of not hurting expected performance. In terms of suggestions, I would recommend a careful rewriting of the paper and its resubmission to a workshop on neural networks compression.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
INTERPRETING CNN COMPRESSION USING INFORMATION BOTTLENECK
In this paper, we first investigate the representation learned in convolutional neural networks at the filter-wise granularity by computing the mutual information between channels of higher conv-layers and input or output variables. Then we identify the approximate minimal sufficient statistics of learned representation based on the information bottleneck principle and propose a novel approach to automatically compress a neural network. This approach prunes a large trained network structurally and automatically by extracting relevant information backpropagately layer by layer in the post-training phase. Our experimental results match the two fundamental data processing inequalities, and prove that mutual information is a fundamental element for examining the efficiency of the internal representations at the filter-wise granularity. In addition, using the information bottleneck principle to interpret structure compression is an efficient method to get closer to the information theoretic limit of compression/prediction problem. Finally, from the observed results, we argue that compression is causally linked to the improved generalization performance.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper proposes a novel way of compressing a CNN by selecting the subset of convolutional filters that maximises the IB criterion. Based on this idea the authors propose an automatic procedure to prune non-informative filters from a trained network and show empirically that, with minimal impact on performance. The idea of the paper is interesting but I feel the relative novelty of the approach is not sufficient to make it eligible as a full conference paper. The idea of feature selection using information gain is a classical approach in standard applied ML literature and the only difference I see in the current approach that it is applied to convolutional filters and not to features themselves. Furthermore, the writing of the paper needs to be revisited, there are many instances of sentences that are confusing or not fully finished such as: - So pruning these high informative filters affects to predict the objects correctly - […] in which network are trained to regularize most of neurons approximately uninformative. - […] then prune the high-informative filters incrementally in the precise of not hurting expected performance. In terms of suggestions, I would recommend a careful rewriting of the paper and its resubmission to a workshop on neural networks compression.
assistant
The paper proposes a novel way of compressing a CNN by selecting the subset of convolutional filters that maximises the IB criterion. Based on this idea the authors propose an automatic procedure to prune non-informative filters from a trained network and show empirically that, with minimal impact on performance. The idea of the paper is interesting but I feel the relative novelty of the approach is not sufficient to make it eligible as a full conference paper. The idea of feature selection using information gain is a classical approach in standard applied ML literature and the only difference I see in the current approach that it is applied to convolutional filters and not to features themselves. Furthermore, the writing of the paper needs to be revisited, there are many instances of sentences that are confusing or not fully finished such as: - So pruning these high informative filters affects to predict the objects correctly - […] in which network are trained to regularize most of neurons approximately uninformative. - […] then prune the high-informative filters incrementally in the precise of not hurting expected performance. In terms of suggestions, I would recommend a careful rewriting of the paper and its resubmission to a workshop on neural networks compression.