HIERARCHICAL INTERPRETATIONS FOR NEURAL NET- WORK PREDICTIONS
Deep neural networks (DNNs) have achieved impressive predictive performance due to their ability to learn complex, non-linear relationships between variables. However, the inability to effectively visualize these relationships has led to DNNs being characterized as black boxes and consequently limited their applications. To ameliorate this problem, we introduce the use of hierarchical interpretations to explain DNN predictions through our proposed method: agglomerative contextual decomposition (ACD). Given a prediction from a trained DNN, ACD produces a hierarchical clustering of the input features, along with the contribution of each cluster to the final prediction. This hierarchy is optimized to identify clusters of features that the DNN learned are predictive. We introduce ACD using examples from Stanford Sentiment Treebank and ImageNet, in order to diagnose incorrect predictions, identify dataset bias, and extract polarizing phrases of varying lengths. Through human experiments, we demonstrate that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs. We also find that ACD's hierarchy is largely robust to adversarial perturbations, implying that it captures fundamental aspects of the input and ignores spurious noise.
This paper proposes a novel approach to explain neural network predictions by learning hierarchical representations of groups of input features and their contribution to the final prediction. The proposed method is a straightforward extension of the contextual decomposition work by (Murdoch et. al. 2018) which estimates feature interpretability for LSTMs. This work extends (Murdoch et. al. '18) to more general NN architectures and further employs agglomerative clustering to identify groups of features-- as opposed to individual features--that are predictive of the output. 
Results are shown using a LSTM trained on the standard Stanford sentiment task and a VCG DNN trained on ImageNet which show the superior performance of the proposed approach. In addition, the paper also provides some survey results where "humans" were asked to pick more interpretable models. 
The paper is nicely written and puts itself nicely in context of the previous work. Though, I have several concerns:
1). Biggest concern: Conditioning on the (Murdoch et. al. 18) paper, the methodological novelty of the proposed approach is minimal. Though, the experimental gains on the vision and NLP tasks are nice.
2). It was unclear to me how the agglomerative algorithm (Algorithm 1) was run. That is, was it run as part of the LSTM estimation for instance for the sentiment task OR was it run post-hoc after getting the model estimates from LSTM? If it was run post-hoc then I am unsure if we can assume that the "agglomeratively grouped CD scores of individual features" are the same as the "CD scores for the groups/interactions of features" in terms of their contribution to the final prediction.
3). Though, the paper mentions several times regarding generalizing (Murdoch et. al. 18) to architectures other than LSTMs but still the experimental results on the sentiment task uses an LSTM as the model. It would have been nice to show the comparative strength of the proposed approach on a different architecture even for the sentiment task. (I understand that the paper uses a different DNN architecture for the vision task).
4). The paper talks several times about diagnosing why a model went wrong e.g. the "negation" in the case of the LSTM model in Figure 2, but never discusses the bigger and more interesting problem. How can we build an improved LSTM model for the sentiment task which classifies that incorrect prediction correctly?