In this paper, we propose a novel approach to interpret a well-trained classification model through systematically investigating effects of its hidden units on prediction making. We search for the core hidden units responsible for predicting inputs as the class of interest under the generative Bayesian inference framework. We model such a process of unit selection as an Indian Buffet Process, and derive a simplified objective function via the MAP asymptotic technique. The induced binary optimization problem is efficiently solved with a continuous relaxation method by attaching a Switch Gate layer to the hidden layers of interest. The resulted interpreter model is thus end-to-end optimized via standard gradient back-propagation. Experiments are conducted with two popular deep convolutional classifiers, respectively well-trained on the MNIST dataset and the CI-FAR10 dataset. The results demonstrate that the proposed interpreter successfully finds the core hidden units most responsible for prediction making. The modified model, only with the selected units activated, can hold correct predictions at a high rate. Besides, this interpreter model is also able to extract the most informative pixels in the images by connecting a Switch Gate layer to the input layer.
Pros
- The paper proposes a novel formulation of the problem of finding hidden units
  that are crucial in making a neural network come up with a certain output.
- The method seems to be work well in terms of isolating a few hidden units that
  need to be kept while preserving classification accuracy.
Cons
- Sections 3.1 and 3.2 are hard to understand. There seem to be inconsistencies
  in the notation. For example,
(1) It would help to clarify whether y^b_n is the prediction score or its
transformation into [0, 1]. The usage is inconsistent.
(2) It is not clear how "y^b_n can be expressed as \sum_{k=1}^K z_{nk}f_k(x_n)"
in general. This is only true for the penultimate layer, and when y^b_n denotes
the input to the output non-linearity. However, this analysis seems to be
applied for any hidden layer and y^b_n is the output of the non-linearity unit
("The new prediction scores are transformed into a scalar ranging from 0 to 1,
denoted as y^b_n.")
(3) Section 3.1 denotes the DNN classifier as F(.), but section 3.2 denotes the
same classifier as f(.).
(4) Why is r_n called the "center" ? I could not understand in what sense is
this the center, and of what ? It seems that the max value has been subtracted
from all the logits into a softmax (which is a fairly standard operation).
- The analysis seems to be about finding neurons that contribute evidence for
  a particular class. This does not address the issue of understanding why the
network makes a certain prediction for a particular input. Therefore this
approach will be of limited use.
- The paper should include more analysis of how this method helps interpret the
  actions of the neural net, once the core units have been identified.
Currently, the focus seems to be on demonstrating that the classifier
performance is maintained as a significant fraction of hidden units are masked.
However, there is not enough analysis on showing whether and how the identified
hidden units help "interpret" the model.
Quality
The idea explored in the paper is interesting and the experiments are described
in enough detail. However, the writing still needs to be polished.
Clarity
The problem formulation and objective function (Section 3.1) was hard to follow.
Originality
This approach to finding important hidden units is novel.
Significance
The paper addresses an important problem of trying to have more interpretable
neural networks. However, it only identifies hidden units that are important for
a class, not what are important for any particular input.  Moreover, the main
thesis of the paper is to describe a method that helps interpret neural network
classifiers. However, the experiments only focus on identifying important hidden
units and fall short of actually providing an interpretation using these hidden
units.