We propose a method which can visually explain the classification decision of deep neural networks (DNNs). There are many proposed methods in machine learning and computer vision seeking to clarify the decision of machine learning black boxes, specifically DNNs. All of these methods try to gain insight into why the network "chose class A" as an answer. Humans, when searching for explanations, ask two types of questions. The first question is, "Why did you choose this answer?" The second question asks, "Why did you not choose answer B over A?" The previously proposed methods are either not able to provide the latter directly or efficiently. We introduce a method capable of answering the second question both directly and efficiently. In this work, we limit the inputs to be images. In general, the proposed method generates explanations in the input space of any model capable of efficient evaluation and gradient evaluation. We provide results, showing the superiority of this approach for gaining insight into the inner representation of machine learning models.
The paper proposes an approach to provide contrastive visual explanations for deep neural networks -- why the network assigned more confidence to some class A as opposed to some other class B. As opposed to the applicability of previous approaches to this problem -- the approach is designed to directly answer the contrastive explanations question rather adapting other visual saliency techniques for the same. Overall, while I find the proposed approach simple -- the paper needs to address some issues regarding the claims made and should provide more quantitative experimental results justifying the same.
- Apart from some flaws in the claims made in the paper, the paper is easy to follow and understand.
- Assuming the availability of a latent model over the images of the input distribution, the proposed approach is directly applicable and faster.
- The authors clearly highlight the problems associated with existing explanation modalities and approaches; ranging from ones applicable to only specific deep architectures to ones using backpropagation based heuristics.
- The proposed approach to generate contrastive explanations is simple and is structured along the lines of methods utilizing probe images to explain decisions -- except for the added advantage that the provided explanations are instance-agnostic due to the assumption of a latent model over the input distribution.
Comments:
- One of the problems highlighted in the paper regarding existing explanation modalities is the use of another black-box to explain the decisions of an existing deep network (also somewhat of a black-box) which the authors claim their model does not suffer from. The proposed approach provides explanations by operating in the latent space of a learned generative model of the input distribution. The learned generator in itself is somewhat of a black-box itself -- there has been prior work indicating how much of the input distribution are GANs able to capture. As such, conditioning on a generative model to propose such contrastive explanations is to some extent using another black-box (generator) to explain the decisions of an existing one. Thus, the above claim made in the paper does not seem well-founded. Furthermore, in experiments, the paper does not provide any quantitatively convincing results to suggest the generator in use is a good one.
- While the authors suggest that a latent model over the input distribution needs to be trained only once and is applicable off-the-shelf for any further contrastive explanations regarding any network operating on the same dataset -- learning such a model of the input space is an overhead in itself. In this light, experiments demonstrating comparisons between GANs and VAEs as the reference generative model for explanations would have made the paper stronger (as the proposed approach relies explicitly on how good the generative model is). 
- The paper proposes an interesting experiment to show that the proposed approach is somewhat capable of capturing slightly adversarial biases in the input domain (adding square to the top-left of images of class ‘8’). While I like this experiment, I feel this has not been explored to completion in the sense of experimenting with robustness with respect to structured as well as unstructured perturbations.
- Typographical Errors: Section 3.1 repeats the use of D for a discriminator as well as the input distribution. Procedure 1 and Procedure 2 share the same titles -- which is slightly misleading. In addition, Procedure 1 is not referenced in the text which makes is hard to understand the utility of the same. In Section 4.1, the use of Gradcam and Lime to generate counterfactual explanations is not very clear and makes it slightly hard to follow. Citations used for Gradcam are wrong -- Sundarajan et al., 2016 should be changed to Selvaraju et al., 2017.
Experimental Issues:
- Experimental results are provided only on MNIST and Fashion-MNIST. Since the paper focuses explicitly on providing contrastive explanations for choosing a class A over another class B -- experiments on datasets which do not have real-images seem insufficient. Additional experiments on at least ImageNet would have made the paper stronger.
Regarding contrastive explanations, experiments on datasets where distractor classes (y_probe) are present in addition to the class interest (y_true) seem important -- PASCAL VOC, COCO, etc. Specifically, since the explanations provided are visual saliency maps the paper would have been stronger if there were experiments suggesting -- what needs to change in a region of an image classified as a ‘cat’ to be classified as a ‘dog’ while there is an instance of the class - ‘dog’ present in the image itself. Also, section 7 in Gradcam (https://arxiv.org/pdf/1610.02391.pdf) provides a procedure to generate counter-factual explanations using Gradcam. Is there a particular reason the authors did not choose to adopt the above technique as a baseline?
- Experimental results provided in the paper are only qualitative -- as such, I do not find the comparisons (and improvements) over the existing approaches convincing enough. Since, there is no clear metric to evaluate contrastive explanations -- human studies to judge the class-discriminativeness (or trust) of the proposed approach would have made the paper stronger.
The authors adressed the issues raised/comments made in the review. In light of my comments below to the author responses -- I am not inclined towards increasing my rating and will stick to my original rating for the paper.