Representer Point Selection for Explaining Deep Neural Networks
We propose to explain the predictions of a deep neural network, by pointing to the set of what we call representer points in the training set, for a given test point prediction. Specifically, we show that we can decompose the pre-activation prediction of a neural network into a linear combination of activations of training points, with the weights corresponding to what we call representer values, which thus capture the importance of that training point on the learned parameters of the network. But it provides a deeper understanding of the network than simply training point influence: with positive representer values corresponding to excitatory training points, and negative values corresponding to inhibitory points, which as we show provides considerably more insight. Our method is also much more scalable, allowing for real-time feedback in a manner not feasible with influence functions.
The manuscript proposes a method to measure the effect that training examples have on the performance of a deep neural network (DNN). Towards this goal, a method is proposed to identify "representer" points that measure the importance that a training image has on the performance of a given model. The paper proposes to identify representer points of two types: i) positive points; which correspond to examples that support the decisions made by the network, and ii) negative points; which inhibit the believe of the model towards certain predictions. Experiments with the VGG-16 and ResNet-50 architectures on the MNIST, CIFAR-10, and Animals with Attributes (AWA) datasets show the strenghts of the proposed method.  The proposed method is sound and well motivated. A detailed formal presentation of the method is provided. In addition, aspects related to the computational cost and numerical properties of the proposed method are adequately discussed.  I find the dataset debugging experiment (Sec.4.1) an interesting means to asses the effect of training examples on the learning of the model being trained.  My main concerns with the manuscript are the following:  i) I consider there is some room for improvement regarding the formal presentation of the method since it can be a bit complex to go through it at the begining. For instance, initially I had troubles understanding what the text "the last intermediate layer feature in the neural network" in l.82 refers to.  ii) While positive/negative representers could be good indicators of the performance of the model, in general, I find hard to find it as a good explanatory means for a specific given test example. For instance, in Fig.4 images of influential examples are presented. However, it is difficult for me to make the direct connection on how these examples "explain" the prediction made for the test image. In my opinion the debugging experiment (Sec.4.1), combined with its related discussion in Sec.4.3 seem to be a more effective use of the proposed method, and perhaps a potentially stronger contribution,  than that of prediction explainability that is claimed by the paper.  iii) Finally, the presented results (Fig. 4,5,6) for the model explanations are mostly qualitative which can be to some extent subjective. In this regard, I find the amount of provided qualitative examples quite reduced. Therefore, I encourage the authors to add supplementary material in order to show extended qualitative results of the explanations produced by their method. Perhaps this can further demonstrate the explanation potential of the proposed method.  I would appreciate if my concerns are addressed in the rebuttal.  ======================== Post rebuttal update ========================  Having read the reviews and the provided rebuttal, my impression of the manuscript is the following.   I do appreciate that a significant part of the rebuttal is focused on addressing my comments. However, I am still not convinced regarding the usefulness of the positive/negative representers as means for explanation. At the end of Sec.1 there is an example highlighting the un-annotated class instances, e.g. antilopes, as negative representers. Later in Sec. 4.2 more examples where the negative representers are images with instances of other classes with similar shape to that of the class of interest are discussed. Similarly, in the rebuttal an example regarding bears was provided. Still very similar to what was presented on the paper.   I would be more convinced about the explanation capabilities of the method if, on top of these representers, there would be an overlaid visualization, e.g. a grad-CAM (Selvaraju et a., arXiv:1610.02391), CNN-Fixation (Reddy et al.,  arXiv:1708.06670 ), DeconvNet (Springenberg et al. ICLR'15), etc. heatmap, highlighting the visual features that make this image an actual negative representer, e.g. highlighting the un-annotated antilopes. At this point, it is the person visualizing the representer negative image, the one coming out with the explanation.  Having said this, and as stated on my initial review, I do find quite valuable the use of these representer points as means for debugging (Sec. 4.1 and Sec 4.3). I share the opinion of the other reviewers that the proposed method holds good promise as a model debugging tool. Moreover, I motivate the authors to update the manuscript to present their method as a debugging method instead of as a tool for explanation. In this regard, I still feel the explanation capability of the proposed method not fully justified.  Finally, I appreciate the additional qualitative examples. I will encourage the authors to enrich their manuscript by adding supplementary material with a good sample of these examples.