Deep Net Triage: Assessing The Criticality of Network Layers by Structural Compression
Deep network compression seeks to reduce the number of parameters in the network while maintaining a certain level of performance. Deep network distillation seeks to train a smaller network that matches soft-max performance of a larger network. While both regimes have led to impressive performance for their respective goals, neither provide insight into the importance of a given layer in the original model, which is useful if we are to improve our understanding of these highly parameterized models. In this paper, we present the concept of deep net triage, which individually assesses small blocks of convolution layers to understand their collective contribution to the overall performance, which we call emph{criticality}. We call it triage because we assess this criticality by answering the question: what is the impact to the health of the overall network if we compress a block of layers into a single layer. We propose a suite of triage methods and compare them on problem spaces of varying complexity. We ultimately show that, across these problem spaces, deep net triage is able to indicate the of relative importance of different layers. Surprisingly, our local structural compression technique also leads to an improvement in overall accuracy when the final model is fine-tuned globally.
 This paper proposes a method to compress a block of layers in a NN (particularly CNNs) evaluates several different sub-approaches. It is easy to follow the paper. Given experiments are nicely setup with intuitive explanations. My main concern about the paper is novelty/importance and lack of experimental comparison to other techniques attacking similar complexity problems. Authors provide a good intro for the problem and existing common approaches addressing network complexity. They locate themselves well in the literature. They provide a simple compact explanation of what their methods are in section 2.3. I find the coding/notation of the sub-methods a little encrypted though. Simple English or better encoding could be used for better readability. Experimental results are explained pretty well. I think the given results/messages are valuable to the community. However the generalizability to different models/network architectures is in question as the paper is mostly based on VGG16. Also the methods/significance seem to be more like an experimental paper and not significant enough for ICLR. I am borderline about this. Comparison between the proposed methods is nicely done but comparison to other literature pruning/compressing models is not experimentally shown. Ideally we expect a reader designing a new architecture to have an idea for which way to go for model compression after reading this paper—results are good to decide between CLRW –STN, but not informative on how they compare to literature. I think this is a significant missing point.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Deep Net Triage: Assessing The Criticality of Network Layers by Structural Compression
Deep network compression seeks to reduce the number of parameters in the network while maintaining a certain level of performance. Deep network distillation seeks to train a smaller network that matches soft-max performance of a larger network. While both regimes have led to impressive performance for their respective goals, neither provide insight into the importance of a given layer in the original model, which is useful if we are to improve our understanding of these highly parameterized models. In this paper, we present the concept of deep net triage, which individually assesses small blocks of convolution layers to understand their collective contribution to the overall performance, which we call emph{criticality}. We call it triage because we assess this criticality by answering the question: what is the impact to the health of the overall network if we compress a block of layers into a single layer. We propose a suite of triage methods and compare them on problem spaces of varying complexity. We ultimately show that, across these problem spaces, deep net triage is able to indicate the of relative importance of different layers. Surprisingly, our local structural compression technique also leads to an improvement in overall accuracy when the final model is fine-tuned globally.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes a method to compress a block of layers in a NN (particularly CNNs) evaluates several different sub-approaches. It is easy to follow the paper. Given experiments are nicely setup with intuitive explanations. My main concern about the paper is novelty/importance and lack of experimental comparison to other techniques attacking similar complexity problems. Authors provide a good intro for the problem and existing common approaches addressing network complexity. They locate themselves well in the literature. They provide a simple compact explanation of what their methods are in section 2.3. I find the coding/notation of the sub-methods a little encrypted though. Simple English or better encoding could be used for better readability. Experimental results are explained pretty well. I think the given results/messages are valuable to the community. However the generalizability to different models/network architectures is in question as the paper is mostly based on VGG16. Also the methods/significance seem to be more like an experimental paper and not significant enough for ICLR. I am borderline about this. Comparison between the proposed methods is nicely done but comparison to other literature pruning/compressing models is not experimentally shown. Ideally we expect a reader designing a new architecture to have an idea for which way to go for model compression after reading this paper—results are good to decide between CLRW –STN, but not informative on how they compare to literature. I think this is a significant missing point.
assistant
 This paper proposes a method to compress a block of layers in a NN (particularly CNNs) evaluates several different sub-approaches. It is easy to follow the paper. Given experiments are nicely setup with intuitive explanations. My main concern about the paper is novelty/importance and lack of experimental comparison to other techniques attacking similar complexity problems. Authors provide a good intro for the problem and existing common approaches addressing network complexity. They locate themselves well in the literature. They provide a simple compact explanation of what their methods are in section 2.3. I find the coding/notation of the sub-methods a little encrypted though. Simple English or better encoding could be used for better readability. Experimental results are explained pretty well. I think the given results/messages are valuable to the community. However the generalizability to different models/network architectures is in question as the paper is mostly based on VGG16. Also the methods/significance seem to be more like an experimental paper and not significant enough for ICLR. I am borderline about this. Comparison between the proposed methods is nicely done but comparison to other literature pruning/compressing models is not experimentally shown. Ideally we expect a reader designing a new architecture to have an idea for which way to go for model compression after reading this paper—results are good to decide between CLRW –STN, but not informative on how they compare to literature. I think this is a significant missing point.