Explaining Neural Networks Semantically and Quantitatively
This paper presents a method to explain the knowledge encoded in a convolutional neural network (CNN) quantitatively and semantically. How to analyze the specific rationale of each prediction made by the CNN presents one of key issues of understanding neural networks, but it is also of significant practical values in certain applications. In this study, we propose to distill knowledge from the CNN into an explainable additive model, so that we can use the explainable model to provide a quantitative explanation for the CNN prediction. We analyze the typical bias-interpreting problem of the explainable model and develop prior losses to guide the learning of the explainable additive model. Experimental results have demonstrated the effectiveness of our method.
 Summary: ========= The paper proposes a technique to explain the embedding space quantitatively and semantically by distilling learned representations into an additive explainable model. Explanation generation for deep neural network models is a growing research area especially for mission critical applications that decision without explanation is unacceptable such as security and medical applications. More importantly, the focus on semantic explanations instead of (in addition to) the usual ‘visual attention or pixel level correlation as explanation’ is interesting. Although, the proposed method is interesting, I have several concerns. I have laid out my comments on the strength of the paper and my concerns as follows. Strength: ======== - Distilling the representation into a separate model to explain rather than incorporating the explanation generation into a single model helps minimize performance degradation due to the additional constraint of generating explanation. - The introduction of semantic explanations is interesting, although the ‘explanation’s are not human-like explanations. See below for further discussion on this. Weakness: ========= - The abstract could use a bit simplification and clarification to make it concise at the same time be inclusive of the key points of the proposed method. More specifically, the second sentence could be simplified to make it easy to follow. - The need for explanation was not strongly motivated in the introduction in the context of existing literature. Existing explanation generating and/or interpretable networks should be discussed and the proposed method should be presented with what gap in existing literature it is trying to address in a more concise form. It is a bit lengthy in current form. - Need to cite related literature on making claims such as “…Distilling knowledge from a pre-trained neural network into an additive model usually suffers from the problem of bias-interpreting.” - The paper mixes interpretation with explanation. Interpretability and explainability are two different things. Invoking attributes of explanations in human cognition, generated explanations should mimic human explanations in that they should be human-understandable, concise, and at the right level of detail. What the introduction promised as semantic explanation was not delivered in the experiments. Explanations for why the network predicts a particular face as attractive are not laundry list of other attributes with positive and negative correlations. Explanations are composed. The authors are learning the inter-attribute relationship (correlations) and this is presented as ‘explanation’. This is disappointing. The title and other earlier discussions need to be revised not to confuse simple interpretation with explanation. - The ‘explanations’ are not as generic as they were advertised earlier. They are in fact specific to specific tasks. It is not obvious from the experiments on object parts and facial attributes tasks how this interpretation using correlation could be generalized to any task learned through a CNN.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Explaining Neural Networks Semantically and Quantitatively
This paper presents a method to explain the knowledge encoded in a convolutional neural network (CNN) quantitatively and semantically. How to analyze the specific rationale of each prediction made by the CNN presents one of key issues of understanding neural networks, but it is also of significant practical values in certain applications. In this study, we propose to distill knowledge from the CNN into an explainable additive model, so that we can use the explainable model to provide a quantitative explanation for the CNN prediction. We analyze the typical bias-interpreting problem of the explainable model and develop prior losses to guide the learning of the explainable additive model. Experimental results have demonstrated the effectiveness of our method.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Summary: ========= The paper proposes a technique to explain the embedding space quantitatively and semantically by distilling learned representations into an additive explainable model. Explanation generation for deep neural network models is a growing research area especially for mission critical applications that decision without explanation is unacceptable such as security and medical applications. More importantly, the focus on semantic explanations instead of (in addition to) the usual ‘visual attention or pixel level correlation as explanation’ is interesting. Although, the proposed method is interesting, I have several concerns. I have laid out my comments on the strength of the paper and my concerns as follows. Strength: ======== - Distilling the representation into a separate model to explain rather than incorporating the explanation generation into a single model helps minimize performance degradation due to the additional constraint of generating explanation. - The introduction of semantic explanations is interesting, although the ‘explanation’s are not human-like explanations. See below for further discussion on this. Weakness: ========= - The abstract could use a bit simplification and clarification to make it concise at the same time be inclusive of the key points of the proposed method. More specifically, the second sentence could be simplified to make it easy to follow. - The need for explanation was not strongly motivated in the introduction in the context of existing literature. Existing explanation generating and/or interpretable networks should be discussed and the proposed method should be presented with what gap in existing literature it is trying to address in a more concise form. It is a bit lengthy in current form. - Need to cite related literature on making claims such as “…Distilling knowledge from a pre-trained neural network into an additive model usually suffers from the problem of bias-interpreting.” - The paper mixes interpretation with explanation. Interpretability and explainability are two different things. Invoking attributes of explanations in human cognition, generated explanations should mimic human explanations in that they should be human-understandable, concise, and at the right level of detail. What the introduction promised as semantic explanation was not delivered in the experiments. Explanations for why the network predicts a particular face as attractive are not laundry list of other attributes with positive and negative correlations. Explanations are composed. The authors are learning the inter-attribute relationship (correlations) and this is presented as ‘explanation’. This is disappointing. The title and other earlier discussions need to be revised not to confuse simple interpretation with explanation. - The ‘explanations’ are not as generic as they were advertised earlier. They are in fact specific to specific tasks. It is not obvious from the experiments on object parts and facial attributes tasks how this interpretation using correlation could be generalized to any task learned through a CNN.
assistant
 Summary: ========= The paper proposes a technique to explain the embedding space quantitatively and semantically by distilling learned representations into an additive explainable model. Explanation generation for deep neural network models is a growing research area especially for mission critical applications that decision without explanation is unacceptable such as security and medical applications. More importantly, the focus on semantic explanations instead of (in addition to) the usual ‘visual attention or pixel level correlation as explanation’ is interesting. Although, the proposed method is interesting, I have several concerns. I have laid out my comments on the strength of the paper and my concerns as follows. Strength: ======== - Distilling the representation into a separate model to explain rather than incorporating the explanation generation into a single model helps minimize performance degradation due to the additional constraint of generating explanation. - The introduction of semantic explanations is interesting, although the ‘explanation’s are not human-like explanations. See below for further discussion on this. Weakness: ========= - The abstract could use a bit simplification and clarification to make it concise at the same time be inclusive of the key points of the proposed method. More specifically, the second sentence could be simplified to make it easy to follow. - The need for explanation was not strongly motivated in the introduction in the context of existing literature. Existing explanation generating and/or interpretable networks should be discussed and the proposed method should be presented with what gap in existing literature it is trying to address in a more concise form. It is a bit lengthy in current form. - Need to cite related literature on making claims such as “…Distilling knowledge from a pre-trained neural network into an additive model usually suffers from the problem of bias-interpreting.” - The paper mixes interpretation with explanation. Interpretability and explainability are two different things. Invoking attributes of explanations in human cognition, generated explanations should mimic human explanations in that they should be human-understandable, concise, and at the right level of detail. What the introduction promised as semantic explanation was not delivered in the experiments. Explanations for why the network predicts a particular face as attractive are not laundry list of other attributes with positive and negative correlations. Explanations are composed. The authors are learning the inter-attribute relationship (correlations) and this is presented as ‘explanation’. This is disappointing. The title and other earlier discussions need to be revised not to confuse simple interpretation with explanation. - The ‘explanations’ are not as generic as they were advertised earlier. They are in fact specific to specific tasks. It is not obvious from the experiments on object parts and facial attributes tasks how this interpretation using correlation could be generalized to any task learned through a CNN.