Probabilistic Connection Importance Inference and Lossless Compression of Deep Neural Networks
Deep neural networks (DNNs) can be huge in size, requiring a considerable a mount of energy and computational resources to operate, which limits their applications in numerous scenarios. It is thus of interest to compress DNNs while maintaining their performance levels. We here propose a probabilistic importance inference approach for pruning DNNs. Specifically, we test the significance of the relevance of a connection in a DNN to the DNN’s outputs using a nonparemtric scoring testand keep only those significant ones. Experimental results show that the proposed approach achieves better lossless compression rates than existing techniques
The paper proposes a new way of evaluating the importance of neural connections which can be used for better model compression. The approach uses a non-parametric statistical test to detect the three way interaction among the two nodes and final output. Some small scale experiments show that the approach achieves better compression rate given the same test error. The approach seems interesting in the sense that, unlike existing techniques, it explicitly measures the three way interaction among the two nodes and the output. Also, it removes the average effects and only considers (non-linear) correlation after removing the mean. The explicit link to the final output and removing average effects allows the method to remove more weights without decreasing the loss by much. However, the drawback of the approach is the significantly increased computation due to 1) quadratic complexity for the kernel methods; 2) unable to cache computation among different pairs of nodes (for example, gradient-based approach can compute importance for all node connections in one forward-backward pass). This limits the applicability of the approach to more interesting cases of larger models (for example, models that work on ImageNet) where model compression is of more urgent need. As a result, the significance and impact of the approach is also limited. The experiment results are interesting in that it shows the proposed approach can achieve better compression rates given the same test error tolerance on a few small datasets. However, as mentioned above, these experiments are less convincing than more complicated models on larger datasets, such as ImageNet where model compression has greater impact. In addition, the test errors on smaller datasets are easier to achieve, sometimes tuning the optimization settings such as learning rates can result in significant improvement. Therefore, such results are more like preliminary. The paper is general clear and well written. The experiment section should include more important information such as what kernel bandwidth is used for Gaussian kernel, which greatly affects performance. Also, the main text introducing the proposed statistical test is a bit verbose, and dense in unnecessary notations. I think it can be made more succinct by presenting a high level idea (removing mean, three way correlations) first and then the final results. Some intermediate results can be put into Appendix.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Probabilistic Connection Importance Inference and Lossless Compression of Deep Neural Networks
Deep neural networks (DNNs) can be huge in size, requiring a considerable a mount of energy and computational resources to operate, which limits their applications in numerous scenarios. It is thus of interest to compress DNNs while maintaining their performance levels. We here propose a probabilistic importance inference approach for pruning DNNs. Specifically, we test the significance of the relevance of a connection in a DNN to the DNN’s outputs using a nonparemtric scoring testand keep only those significant ones. Experimental results show that the proposed approach achieves better lossless compression rates than existing techniques
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper proposes a new way of evaluating the importance of neural connections which can be used for better model compression. The approach uses a non-parametric statistical test to detect the three way interaction among the two nodes and final output. Some small scale experiments show that the approach achieves better compression rate given the same test error. The approach seems interesting in the sense that, unlike existing techniques, it explicitly measures the three way interaction among the two nodes and the output. Also, it removes the average effects and only considers (non-linear) correlation after removing the mean. The explicit link to the final output and removing average effects allows the method to remove more weights without decreasing the loss by much. However, the drawback of the approach is the significantly increased computation due to 1) quadratic complexity for the kernel methods; 2) unable to cache computation among different pairs of nodes (for example, gradient-based approach can compute importance for all node connections in one forward-backward pass). This limits the applicability of the approach to more interesting cases of larger models (for example, models that work on ImageNet) where model compression is of more urgent need. As a result, the significance and impact of the approach is also limited. The experiment results are interesting in that it shows the proposed approach can achieve better compression rates given the same test error tolerance on a few small datasets. However, as mentioned above, these experiments are less convincing than more complicated models on larger datasets, such as ImageNet where model compression has greater impact. In addition, the test errors on smaller datasets are easier to achieve, sometimes tuning the optimization settings such as learning rates can result in significant improvement. Therefore, such results are more like preliminary. The paper is general clear and well written. The experiment section should include more important information such as what kernel bandwidth is used for Gaussian kernel, which greatly affects performance. Also, the main text introducing the proposed statistical test is a bit verbose, and dense in unnecessary notations. I think it can be made more succinct by presenting a high level idea (removing mean, three way correlations) first and then the final results. Some intermediate results can be put into Appendix.
assistant
The paper proposes a new way of evaluating the importance of neural connections which can be used for better model compression. The approach uses a non-parametric statistical test to detect the three way interaction among the two nodes and final output. Some small scale experiments show that the approach achieves better compression rate given the same test error. The approach seems interesting in the sense that, unlike existing techniques, it explicitly measures the three way interaction among the two nodes and the output. Also, it removes the average effects and only considers (non-linear) correlation after removing the mean. The explicit link to the final output and removing average effects allows the method to remove more weights without decreasing the loss by much. However, the drawback of the approach is the significantly increased computation due to 1) quadratic complexity for the kernel methods; 2) unable to cache computation among different pairs of nodes (for example, gradient-based approach can compute importance for all node connections in one forward-backward pass). This limits the applicability of the approach to more interesting cases of larger models (for example, models that work on ImageNet) where model compression is of more urgent need. As a result, the significance and impact of the approach is also limited. The experiment results are interesting in that it shows the proposed approach can achieve better compression rates given the same test error tolerance on a few small datasets. However, as mentioned above, these experiments are less convincing than more complicated models on larger datasets, such as ImageNet where model compression has greater impact. In addition, the test errors on smaller datasets are easier to achieve, sometimes tuning the optimization settings such as learning rates can result in significant improvement. Therefore, such results are more like preliminary. The paper is general clear and well written. The experiment section should include more important information such as what kernel bandwidth is used for Gaussian kernel, which greatly affects performance. Also, the main text introducing the proposed statistical test is a bit verbose, and dense in unnecessary notations. I think it can be made more succinct by presenting a high level idea (removing mean, three way correlations) first and then the final results. Some intermediate results can be put into Appendix.