Interpreting Deep Classification Models With Bayesian Inference
In this paper, we propose a novel approach to interpret a well-trained classification model through systematically investigating effects of its hidden units on prediction making. We search for the core hidden units responsible for predicting inputs as the class of interest under the generative Bayesian inference framework. We model such a process of unit selection as an Indian Buffet Process, and derive a simplified objective function via the MAP asymptotic technique. The induced binary optimization problem is efficiently solved with a continuous relaxation method by attaching a Switch Gate layer to the hidden layers of interest. The resulted interpreter model is thus end-to-end optimized via standard gradient back-propagation. Experiments are conducted with two popular deep convolutional classifiers, respectively well-trained on the MNIST dataset and the CI- FAR10 dataset. The results demonstrate that the proposed interpreter successfully finds the core hidden units most responsible for prediction making. The modified model, only with the selected units activated, can hold correct predictions at a high rate. Besides, this interpreter model is also able to extract the most informative pixels in the images by connecting a Switch Gate layer to the input layer.
 The paper develops a technique to understand what nodes in a neural network are important for prediction. The approach they develop consists of using an Indian Buffet Process to model a binary activation matrix with number of rows equal to the number of examples. The binary variables are estimated by taking a relaxed version of the asymptotic MAP objective for this problem. One question from the use of the Indian Buffet Process: how do the asymptotics of the feature allocation determine the number of hidden units selected? Overall, the results didn*t warrant the complexity of the method. The results are neat, but I couldn*t tell why this approach was better than others. Lastly, can you intuitively explain the additivity assumption in the distribution for p(y*)
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Interpreting Deep Classification Models With Bayesian Inference
In this paper, we propose a novel approach to interpret a well-trained classification model through systematically investigating effects of its hidden units on prediction making. We search for the core hidden units responsible for predicting inputs as the class of interest under the generative Bayesian inference framework. We model such a process of unit selection as an Indian Buffet Process, and derive a simplified objective function via the MAP asymptotic technique. The induced binary optimization problem is efficiently solved with a continuous relaxation method by attaching a Switch Gate layer to the hidden layers of interest. The resulted interpreter model is thus end-to-end optimized via standard gradient back-propagation. Experiments are conducted with two popular deep convolutional classifiers, respectively well-trained on the MNIST dataset and the CI- FAR10 dataset. The results demonstrate that the proposed interpreter successfully finds the core hidden units most responsible for prediction making. The modified model, only with the selected units activated, can hold correct predictions at a high rate. Besides, this interpreter model is also able to extract the most informative pixels in the images by connecting a Switch Gate layer to the input layer.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper develops a technique to understand what nodes in a neural network are important for prediction. The approach they develop consists of using an Indian Buffet Process to model a binary activation matrix with number of rows equal to the number of examples. The binary variables are estimated by taking a relaxed version of the asymptotic MAP objective for this problem. One question from the use of the Indian Buffet Process: how do the asymptotics of the feature allocation determine the number of hidden units selected? Overall, the results didn*t warrant the complexity of the method. The results are neat, but I couldn*t tell why this approach was better than others. Lastly, can you intuitively explain the additivity assumption in the distribution for p(y*)
assistant
 The paper develops a technique to understand what nodes in a neural network are important for prediction. The approach they develop consists of using an Indian Buffet Process to model a binary activation matrix with number of rows equal to the number of examples. The binary variables are estimated by taking a relaxed version of the asymptotic MAP objective for this problem. One question from the use of the Indian Buffet Process: how do the asymptotics of the feature allocation determine the number of hidden units selected? Overall, the results didn*t warrant the complexity of the method. The results are neat, but I couldn*t tell why this approach was better than others. Lastly, can you intuitively explain the additivity assumption in the distribution for p(y*)