Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives
In this paper we propose a novel method that provides contrastive explanations justifying the classification of an input by a black box classifier such as a deep neural network. Given an input we find what should be minimally and sufficiently present (viz. important object pixels in an image) to justify its classification and analogously what should be minimally and necessarily absent (viz. certain background pixels). We argue that such explanations are natural for humans and are used commonly in domains such as health care and criminology. What is minimally but critically absent is an important part of an explanation, which to the best of our knowledge, has not been explicitly identified by current explanation methods that explain predictions of neural networks. We validate our approach on three real datasets obtained from diverse domains; namely, a handwritten digits dataset MNIST, a large procurement fraud dataset and a brain activity strength dataset. In all three cases, we witness the power of our approach in generating precise explanations that are also easy for human experts to understand and evaluate.
The paper proposes a Contrastive Explanation Method (CEM) with the goal of justifying, or explaining, the decisions made by deep neural networks. A notable characteristic of the proposed method is that for every class of interest it aims at identifying two sets of features that can be used to jutify the decisions related to a given class. The first type, pertinent positives (PN), are features whose occurrence are the minimum necessary to justify the classification prediction.  The second type, pertinent negatives (PN) are features whose absence is necessary to justify the prediction of a given class. Given an input image, explanations are given by extending the output by highlighting these two types of features.    Experiments on the MNIST [38], ABIDE-I [11] and a preocurement fraud dataset [9] show the performance of the proposed method.  ===================  Overall the content of the manuscript is sound, clear and easy to follow.  I find valuable the fact that the evaluation includes experiments datasets on multiples data modalities, i.e. visual data (Sec. 4.1 and 4.3) and text-based data (Sec. 4.2). In addition, this seems to be one of the first works focusing  explictly on the identification of pertinent negatives.  In addition, there seems to be code available in the supplementary material. If the plan is to release code to run the proposed method, this could be good for reproducibility of the presented results.  Having said this, my concerns with the manuscript are the following:   Even when I find novel the fact that the proposed method is explictly designed to identify and indicate PN features, this is not the first work to highlight this type of negative feature. Examples of existing works include Samek et al., 2017 and more recently, Oramas et al. arXiv:1712.06302, which as part of their generated output provided some explanations supporting the absence of specific features. I consider positioning with respect to these works a must in order to effectively highlight the novelty brought by the proposed method.  In Section 3, it is stated that the proposed method aims at identify the features that could serve as PPs and PNs for model explanation.  However, in several of the figures/Tables displaying the explanations of the proposed method just one output of each type of feature (PP or PN) is given. To these outputs already highlight more than one feature of each type? if yes, how can we distinguish between the different features highlighted within a type.  if not, an multiple outputs are posible, what is the criterion to select which features to highlight as part of the output?  In Section 3 it is mentioned the need for an autoencoder (AE). DUring the evaluation, this autoencoder is only employed in 2/3 of the conducted experiments. At this point it is not clear to me when this autoencoder is necessary or not. Is there a principled means to verify whether this autoencoder is necessary?  Experiments are conducted on relatively simple and small datasets. Moreover, the Procurement Fraud dataset used in Section 4.2 does not seem to be public. If this is indeed the case, this would limit future comparisons with respect to this work.  In addition, while the manuscript stresses in the abstract the black box characteristic of deep neural networks (DNNs), the tested architectures are relatively simple and shallow. Therefore the potential of the proposed method is unclear in more complex scenarios. For the case of DNNs for visual data, I would suggest evaluating on one of the standard architectures, e.g. alexnet, VGG, ResNet, etc., for which pre-trained models can be found online. Results on one of this deeper more complex settings will strengthen the potential of the proposed method.  Finally, the presented results are mostly qualitative.  In this regard I would suggest performing a either: a) an occlussion analysis(Zeiler et al., ECCV 2014, Samek et al., 2017) , b) a pointing experiment (Zhang et al., ECCV 2016) or, c) a measurement of explanation accuracy by feature coverage (Oramas et al. arXiv:1712.06302).  Any of these protocols should allow to quantitatively evaluate the relevance of the features highlighted by the proposed as part of the explanation. In addition, I find the set of provided qualitative examples quite reduced. In this regard, I encourage the authors to update the supplementary material in order to show extended qualitative results of the explanations produced by their method.  I would appreciate if my concerns are addressed in the rebuttal.   ========================== Full references ==========================    Jose Oramas, Kaili Wang, Tinne Tuytelaars "Visual Explanation by Interpretation: Improving Visual Feedback Capabilities of Deep Neural Networks" (2017), arXiv:1712.06302.  Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller: "Evaluating the Visualization of What a Deep Neural Network has Learned" IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660-2673, 2017  Matthew D. Zeiler and Rob Fergus.  "Visualizing and understanding convolutional networks" ECCV, 2014.  Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen, Stan Sclaroff "Top-down Neural Attention by Excitation Backprop" ECCV 2016.    ======================================== Post Rebuttal Update ========================================  I thank the authors for their rebuttal. The provided rebuttal satisfied to some extent most of the points I raised on my review.  I share the R1's view that the proposed idea is interesting. Though, I do not consider the fact of highlighting features that will produce a negative effect that novel. Indeed, most current work in interpretable machine learning (e.g. Selvaraju et al., arXiv:1610.02391,  Chattopadhay et al., WACV'18, Reddy et al., arXiv:1708.06670 ) only highlights features that are present in the instance. However, highlighting other, potentially conflicting, features related to other classes will mostly require generating such visualizations for other non-max classes with high score for the instance being explained.  I share the concerns of R1 on that the evaluation aspect could be improved. Currently, results are reported mostly on a  qualitative fashion. While some deterministic behavior is expected when giving the PPs and PNs as part of the input to the neural network, the identification of these PPs and PNs is not perfect. I still would like to see empirical (quantitative) evidence of to what these elements satisfy that expected behavior.  Moreover, for the case of the MNIST experiment, the rebuttal (responding to R1) states that there is no widely accepted quantitative analysis for measuring explanations on visual datasets. Indeed, there is no unique metric for this yet. However, as I stated on my review, there are several evaluation methods that have been proposed and can help position the proposed method appropriately.   At this point, the rebuttal does not change my initial opinion of the manuscript. I still consider the manuscript a good paper and a good match for NIPS.