Deep neural networks have achieved impressive performance in handling complicated semantics in natural language, while mostly treated as black boxes. To explain how the model handles compositional semantics of words and phrases, we study the hierarchical explanation problem. We highlight the key challenge is to compute non-additive and context-independent importance for individual words and phrases. We show some prior efforts on hierarchical explanations, e.g. contextual decomposition, do not satisfy the desired properties mathematically, leading to inconsistent explanation quality in different models. In this paper, we propose a formal way to quantify the importance of each word or phrase to generate hierarchical explanations. We modify contextual decomposition algorithms according to our formulation, and propose a model-agnostic explanation algorithm with competitive performance. Human evaluation and automatic metrics evaluation on both LSTM models and fine-tuned BERT Transformer models on multiple datasets show that our algorithms robustly outperform prior works on hierarchical explanations. We show our algorithms help explain compositionality of semantics, extract classification rules, and improve human trust of models 1 .
Summary - The paper addresses the problem of hierarchical explanations in deep models that handle compositional semantics of words and phrases. The paper first highlights desirable properties for importance attribution scores in hierarchical explanations, specifically, non-additivity and context independence, and shows how prior work on additive feature attribution and context decomposition doesn’t accurately capture these notions. After highlighting the said properties in context of related work, the authors propose an approach to calculate the context-independent importance of a phrase by computing the difference in scores with and without masking out the phrase marginalized over all possible surrounding word contexts (approximated by sampling surrounding context for a fixed radius under a language model). Furthermore, based on the above, the authors propose two more score attribution approaches -- based on integrating the above sampling step with (1) the contextual decomposition pipeline and (2) the input occlusion pipeline. Experimentally, the authors find that the attribution scores assigned by the proposed approach are more correlated with human annotations compared to prior approaches and additionally, the generated explanations turn out to be more trustworthy when humans evaluate their quality.
Strengths
- The paper is well-written and generally easy to follow. The authors do a good job of motivating and highlighting the desired properties of importance attribution scores and developing the proposed scoring mechanism. The proposed scoring mechanism ties in seamlessly with the existing contextual decomposition and occlusion pipelines and leads to improved performance when the generated explanations are evaluated.
- The proposed approach involving masking out the phrase and marginalizing over possible surrounding word-concepts is novel and offers an interesting perspective on how to approach context independent scoring of phrases -- (1) phrases don’t exist independent of the surrounding context and therefore marginalizing over all possible surrounding concepts makes sense and (2) replacing the intractable enumeration over all possible surrounding concepts with samples from a language model makes the score attribution process faster and more scalable modulo the learnt language model.
- Sec. 4.4 offers interesting insights. I like that the authors performed this ablation given that the expectation over surrounding contexts is computed approximately via samples under a language model. There’s a clear increase in terms of the attribution scores as the number of samples increases and the neighborhood size is increased. It is interesting to note that there is an approaching plateau region where increasing the neighborhood size won’t affect the assigned scores. This experiment provides a holistic picture of the behavior of the interpretability toolkit (manifesting in terms of attribution scores) given the approximations involved. I would encourage the authors to flesh this out even more. 
Weaknesses
Having said that, there are some minor comments that I’d like to point out / get the authors’ opinion on. Highlighting these below:
- While SOC and SCD don’t always end up outperforming other approaches (specifically Statistic) on the SST-2, Yelp and TACRED datasets (Table. 1), for the human evaluation experiments, the authors only compare with CD, Direct Feed, ACD and GradSHAP. Do the authors have any insights on how well does Statistic perform on the human-evaluation set of experiments?
- While inspiring trust in users is one aspect of evaluating explanations via humans, it’s slightly unclear what ‘trust’ in this sense inherently identifies. Although, it might implicitly capture some notion of reliability (and predictability of the explanations by humans), asking users to rank explanations across a spectrum of ‘best’ to ‘worst’ doesn’t explicitly capture that. Another possible aspect to look into could be -- ‘’Do the generated explanations help humans predict the output of the model?’’ This captures reliability in a very explicit sense. Do the authors have any thoughts on this and potential experiments that might address this? I don’t think not addressing this is necessarily detrimental to the paper but I’m curious to hear the thoughts of the authors on the same. 
Reasons for rating
Beyond the above points of discussion, I don’t have major weaknesses to point 	out. I generally like the paper. The authors do a good job of identifying the sliver in which they make their contribution and motivate the same appropriately. The proposed phrase attribution scoring mechanism is motivated from a novel perspective and has a reasonable approximation characterized appropriately by the ablations performed. The strengths and weaknesses highlighted above form the basis of my rating.