VISUAL EXPLANATION BY INTERPRETATION: IMPROVING VISUAL FEEDBACK CAPABILITIES OF DEEP NEURAL NETWORKS
Interpretation and explanation of deep models is critical towards wide adoption of systems that rely on them. In this paper, we propose a novel scheme for both interpretation as well as explanation in which, given a pretrained model, we automatically identify internal features relevant for the set of classes considered by the model, without relying on additional annotations. We interpret the model through average visualizations of this reduced set of features. Then, at test time, we explain the network prediction by accompanying the predicted class label with supporting visualizations derived from the identified features. In addition, we propose a method to address the artifacts introduced by strided operations in deconvNetbased visualizations. Moreover, we introduce an8Flower, a dataset specifically designed for objective quantitative evaluation of methods for visual explanation. Experiments on the MNIST, ILSVRC12, Fashion144k and an8Flower datasets show that our method produces detailed explanations with good coverage of relevant features of the classes of interest.
Summary: the paper proposes a method for Deep Neural Networks (DNN) that identifies automatically relevant features of the set of the classes, enriching the predictions made with the visual features that contributed to that class, supporting, thus, interpretation (understanding what the model has learned) and explanation (justification of the predictions/classifications made by the model). This scheme does not rely on additional annotations, like earlier techniques do.
The contributions of this paper are relevant to, I would say, a large segment of the AI community, since interpretability and explainability of AI (XAI) is the focus of many current works in the area, and there are still many unresolved issues. I consider this paper suitable for ICLR 2019, in particular, it fits the call for papers topic “visualization or interpretation of learned representations”.
The authors also present a new dataset (am8Flower) that can be used by the community for future evaluations of explanation methods for DNN. From my point of view, this is a significant contribution, since there is a lack of datasets that can be used for evaluation.
The authors motivate properly the need for this research/study, addressing the main weakness of the two more common strategies for interpreting DNN, (1) manually inspecting visualizations of every single filter or (2) comparing the internal activations produced by a given model w.r.t. a dataset with pixel-wise annotations of possibly relevant concepts.
I would encourage the authors to write the limitations and weakness of their proposal w.r.t. similar approaches they reviewed. I am aware that the space is limited, but in p.8, section 4.3, when Table 1 is introduced and the authors confirm that their proposal has higher IoU than other methods, the authors could explain, in brief, what are the weaknesses of their method w.r.t. the other approaches analyzed.
Another clarification concerns the initialization of input parameters, such as sparsity; e.g., p.6 sparsity is initialized with 10 for all datasets, why? How has this value been selected and how sensitive is the performance regarding variations of this value?
Once again, I know that the space is limited, but I would like to be able to see some of the figures better (since this is an essential part of the paper). The additional material complements very well the paper and shows larger figures, but I think that the paper itself should be self-sufficient, and figures like Fig. 5 should be enlarged so it is easier to see some details.
Just a concern or something that I quite did not understand about one of the arguments the authors use to justify the evaluation carried out: the authors claim that they want to avoid the subjectivity introduced by humans (citing Gonzalez-Garcia et al. 2017), and prefer to avoid user studies, presenting a more objective approach in their evaluation. Ok, but then, the analysis presented in, for example, page 7, is based mainly in their interpretation of the results, a qualitative analysis of the images (we can see fur patterns, this and that, etc.). So aren’t they interpreting the results obtained as users? So after all, aren’t the visual explanations and feedback intended for users? Why should we claim that we want to avoid the subjectivity introduced by humans in the evaluation when the method proposed here is actually going to be used by users –with their inherent subjectivity? I do not mean that the evaluation carried out is not interesting per se, but it could be motivated differently, or it could be complemented later on with future user studies (that would make an interesting addition to the paper). Moreover, I also wonder whom the authors see as intended users for the proposed scheme.
Small comments:
P.1 “useful insights on the internal representations”  insights into the internal representations.
P. 2: space needed in “back-propagation methods.Third,”
P. 3: Remove “s” in verb (plural authors): “Similarly, Bach et al. (2015) decomposes the classification”  decompose or decomposed
P.3: n needed “Chattopadhyay et al. (2018) exteded”  extended
P.3: “This saliency-based protocol assume that”  protocol assumes
P.3: “highlighted by the the explanations”  remove one “the”
P. 5: “space. As as result we get”  remove one “as”
P. 5: “and compensate this change”  compensate for this change
P. 6: “In this experiment we verify”  In this experiment, we verify
P. 6: “To this end, given a set of identified features we”  To this end, given a set of identified features, we
P. 6: “Note that the OnlyConv method, makes the assumption”  remove “,” after method
P. 7: “In order to get a qualitative insight of the type of”  insight into the
P. 7: I would write siamese and persian cat with capital “S” and “P” (Siamese, Persian)
P. 7: others/ upper “Some focus on legs, covered and uncovered, while other focus on the upped body part.”  while others focus on the upper body part
P. 7: “These visualizations answers the question”  answer
P. 7:  “In this section we assess”  In this section, we
P. 7: Plural “We show these visualization for different”  these visualizations
P. 7: In “Here our method reaches a mean difference on prediction confidence”  difference in prediction …
P. 7: “This suggest that our method is able”  This suggests that
P. 8: state-of-the-art
P. 8: “has higher mean IoU”  has a higher mean IoU
Whole document: when using “i.e.” add “,” after: i.e.,
References: Some of the references in the list have very little information to be able to find it/proper academic citation, e.g. , Yosinski et al. 2015; Vedaldi and Lenc, 2015:
Jason Yosinski, Jeff Clune, Anh Mai Nguyen, Thomas J. Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. 2015.
A. Vedaldi and K. Lenc. Matconvnet: Convolutional neural networks for matlab. In MM, 2015.
Ref Doersch et al.: What makes paris look like paris?  Paris