Attention mechanisms have seen wide adoption in neural NLP models. In addition to improving predictive performance, these are often touted as affording transparency: models equipped with attention provide a distribution over attended-to input units, and this is often presented (at least implicitly) as communicating the relative importance of inputs. However, it is unclear what relationship exists between attention weights and model outputs. In this work we perform extensive experiments across a variety of NLP tasks that aim to assess the degree to which attention weights provide meaningful explanations" for predictions. We find that they largely do not. For example, learned attention weights are frequently uncorrelated with gradient-based measures of feature importance, and one can identify very different attention distributions that nonetheless yield equivalent predictions. Our findings show that standard attention modules do not provide meaningful explanations and should not be treated as though they do. Code to reproduce all experiments is available at https://github.com/successar/ AttentionExplanation . 1 Introduction and Motivation Attention mechanisms (Bahdanau et al., 2014) induce conditional distributions over input units to compose a weighted context vector for downstream modules. These are now a near-ubiquitous component of neural NLP architectures. Attention weights are often claimed (implicitly or explicitly) to afford insights into the inner-workings of models: for a given output one can inspect the inputs to which the model assigned large attention weights. Li et al. (2016) summarized this commonly held view in NLP: Attention provides an important way to explain the workings of neural models". Indeed, claims that attention provides after 15 minutes watching the movie i was asking myself what to do leave the theater sleep or try to keep watching the movie to see if there was anything worth i finally watched the movie what a waste of time maybe i am not a 5 years old kid anymore original adversarial after 15 minutes watching the movie i was asking myself what to do leave the theater sleep or try to keep watching the movie to see if there was anything worth i finally watched the movie what a waste of time maybe i am not a 5 years old kid anymore f ( x | , ) = 0 . interpretability are common in the literature, e.g., (Xu et al., 2015; Choi et al., 2016; Lei et al., 2017; Martins and Astudillo, 2016; Xie et al., 2017). 1 Implicit in this is the assumption that the input units (e.g., words) accorded high attention weights are responsible for model outputs. But as far as we are aware, this assumption has not been formally evaluated, and our findings here suggest that it is problematic. More specifically, we empirically investigate the relationship between attention weights, inputs, and outputs. Assuming attention provides an explanation for model predictions, we might expect the following properties to hold. (i) Attention weights should correlate with feature importance measures (e.g., gradient-based measures); (ii) Alternative (or counterfactual ) attention weight configurations ought to yield corresponding changes in prediction (and if they do not then are equally plausible as explanations). We report that neither property is consistently observed by standard attention mechanisms in the context of text classification, question answering (QA), and Natural Language Inference (NLI) tasks when RNN encoders are used. 1 We do not intend to single out any particular work; indeed one of the authors has himself presented (supervised) attention as providing interpretability (Zhang et al., 2016). Consider Figure 1. The left panel shows the original attention distribution  over the words of a particular movie review using a standard attentive BiLSTM architecture for sentiment analysis. It is tempting to conclude from this that the token waste is largely responsible for the model coming to its disposition of negative' (  y = 0 . 01 ). But one can construct an alternative attention distribution   (right panel) that attends to entirely different tokens yet yields an essentially identical prediction (holding all other parameters of f ,  , constant). Such counterfactual distributions imply that explaining the original prediction by highlighting attended-to tokens is misleading insofar as alternative attention distributions would have yielded an equivalent prediction (e.g., one might conclude from the right panel that model output was due primarily to was rather than waste ). Further, the attention weights in this case correlate only weakly with gradient-based measures of feature importance (  g = 0 . 29 ). And arbitrarily permuting the entries in  yields a median output difference of 0.006 with the original prediction. These and similar findings call into question the view that attention provides meaningful insight into model predictions. We thus caution against using attention weights to highlight input tokens responsible for model outputs and constructing just-so stories on this basis, particularly with complex encoders. Research questions and contributions . We examine the extent to which the (often implicit) narrative that attention provides model transparency 2 holds across tasks by exploring the following empirical questions. 1. To what extent do induced attention weights correlate with measures of feature importance  specifically, those resulting from gradients and leave-one-out methods? 2. Would alternative attention weights (and hence distinct heatmaps/explanations) necessarily yield different predictions? Our findings with respect to these questions (as-suming a BiRNN encoder) are summarized as follows: (1) Only weakly and inconsistently, and, (2) No; it is very often possible to construct adversarial attention distributions that yield effectively 2 Defined as per (Lipton, 2016); we are interested in whether attended-to features are responsible for outputs. equivalent predictions as when using the originally induced attention weights, despite attending to entirely different input features. Further, randomly permuting attention weights often induces only minimal changes in output. We consider exemplar NLP tasks for which attention mechanisms are commonly used: classification, natural language inference (NLI), and question answering. 3 We adopt the following general modeling assumptions and notation. We assume model inputs x  RT | V | , composed of one-hot encoded words at each position. These are passed through an embedding matrix E which provides dense ( d dimensional) token representations x e  RT  d . Next, an encoder Enc consumes the embedded tokens in order, producing T m -dimensional hidden states: h = Enc ( x e )  RT  m . We predominantly consider a Bi-RNN as the encoder module, but for completeness we also analyze convolutional and (unordered) aver-age embedding' variants. 4 A similarity function  maps h and a query Q  R m (e.g., hidden representation of a question in QA, or the hypothesis in NLI) to scalar scores, and attention is then induced over these:   = softmax (  ( h , Q ))  RT . In this work we consider two common similarity functions: Additive  ( h , Q ) = v T tanh ( W 1 h + W 2 Q ) (Bahdanau et al., 2014) and Scaled Dot-Product  ( h , Q ) = hQ  m (Vaswani et al., 2017), where v , W 1 , W 2 are model parameters. Finally, a dense layer Dec with parameters  consumes a weighted instance representation and yields a prediction  y =  (   h  )  R |Y| , where h  = (cid:80) Tt =1   t  h t ;  is an output activation function; and |Y| denotes the label set size. For binary text classification , we use: Stanford Sentiment Treebank (SST) (Socher et al., 2013). 10,662 sentences tagged with sentiment on a scale from 1 (most negative) to 5 (most positive). We filter out neutral instances and dichotomize the remaining sentences into positive (4, 5) and negative (1, 2). IMDB Large Movie Reviews Corpus (Maas et al., 2011). Binary sentiment classification dataset containing 50,000 polarized (positive or negative) movie reviews, split into half for training and testing. Twitter Adverse Drug Reaction dataset (Nikfar-jam et al., 2015). A corpus of  8000 tweets retrieved from Twitter, annotated by domain experts as mentioning adverse drug reactions. 20 Newsgroups (Hockey vs Baseball) . Collection of  20,000 newsgroup correspondences, partitioned (nearly) evenly across 20 categories. We extract instances belonging to baseball and hockey , which we designate as 0 and 1, respectively, to derive a binary classification task. 5 496,835 news articles from 2000+ sources. AG News Corpus (Business vs World) . We follow (Zhang et al., 2015) in filtering out all but the top 4 categories. We consider the binary classification task of discriminating between world (0) and business (1) articles. MIMIC ICD9 (Diabetes) (Johnson et al., 2016). A subset of discharge summaries from the MIMIC III dataset of electronic health records. The task is to recognize if a given summary has been labeled with the ICD9 code for diabetes (or not). MIMIC ICD9 (Chronic vs Acute Anemia) (John-son et al., 2016). A subset of discharge summaries from MIMIC III dataset (Johnson et al., 2016) known to correspond to patients with anemia. Here the task to distinguish the type of anemia for each report  acute (0) or chronic (1). matic parsing of news articles from CNN. Each instance comprises a paragraph-question-answer triplet, where the answer is one of the anonymized entities in the paragraph. bAbI (Weston et al., 2015). We consider the three tasks presented in the original bAbI dataset paper, training separate models for each. These entail finding (i) a single supporting fact for a question and (ii) two or (iii) three supporting statements, chained together to compose a coherent line of reasoning. The SNLI dataset (Bowman et al., 2015). 570k human-written English sentence pairs manually labeled for balanced classification with the labels neutral , contradiction , and entailment , supporting the task of natural language inference (NLI). In this work, we generate an attention distribution over premise words conditioned on the hidden representation induced for the hypothesis. We restrict ourselves to comparatively simple instantiations of attention mechanisms, as described in the preceding section. This means we do not consider recently proposed BiAttentive' architectures that attend to tokens in the respective inputs, conditioned on the other inputs (Parikh et al., 2016; Seo et al., 2016; Xiong et al., 2016). Table 1 provides summary statistics for all datasets, as well as the observed test performances for additional context. We run a battery of experiments that aim to examine empirical properties of learned attention weights and to interrogate their interpretability and transparency. The key questions are: Do Gradient (BiLSTM)  g Gradient (Average)  g Leave-One-Out (BiLSTM)  loo Dataset Class Mean  Std. learned attention weights agree with alternative, natural measures of feature importance ? And, Had we attended to different features, would the prediction have been different ? More specifically, in Section 4.1, we empirically analyze the correlation between gradient-based feature importance and learned attention weights, and between leave-one-out' (LOO) measures and the same. In Section 4.2 we then consider counterfactual (to those observed) attention distributions. Under the assumption that attention weights are explanatory, such counterfactual distributions may be viewed as alternative potential explanations; if these do not correspondingly change model output, then the original attention weights do not provide unique explanation for predictions, i.e., attending to other features could have resulted in the same output. To generate counterfactual attention distributions, we first consider randomly permuting observed attention weights and recording associated changes in model outputs (4.2.1). We then propose explicitly searching for adversarial attention weights that maximally differ from the observed attention weights (which one might show in a heatmap and use to explain a model prediction), and yet yield an effectively equivalent prediction (4.2.2). The latter strategy also provides a useful potential metric for the reliability of attention weights as explanations: we can report a measure quantifying how different attention weights can be for a given instance without changing the model output by more than some threshold (cid:15) . All results presented below are generated on test sets. We present results for Additive attention below. The results for Scaled Dot Product in its place are comparable. We provide a web interface to interactively browse the (very large set of) plots for all datasets, model variants, and experiment types: https://successar.github. io/AttentionExplanation/docs/ . In the following sections, we use Total Variation Distance (TVD) as the measure of change between output distributions, defined as follows. TVD ( y 1 ,  y 2 ) = 12 (cid:80) |Y| i =1 |  y 1 i   y 2 i | . We use the Jensen-Shannon Divergence (JSD) to quantify the difference between two attention distributions: JSD (  1 ,  2 ) = 12 KL [  1 ||  1 +  2 2 ] + 12 KL [  2 ||  1 +  2 2 ] . We empirically characterize the relationship between attention weights and corresponding feature importance scores. Specifically we measure correlations between attention and: (1) gradient based measures of feature importance (  g ), and, (2) differences in model output induced by leaving features out (  loo ). While these measures are themselves insufficient for interpretation of neu-1.0 0.5 0.0 0.5 1.0 0.000 0.025 0.050 0.075 0.100 0.125 ral model behavior (Feng et al., 2018), they do provide measures of individual feature importance with known semantics (Ross et al., 2017). It is thus instructive to ask whether these measures correlate with attention weights. The process we follow to quantify this is described in Algorithm 1. We denote the input resulting from removing the word at position t in x by x  t . Note that we disconnect the computation graph at the attention module so that the gradient does not flow through this layer. Table 2 reports summary statistics of Kendall  correlations for each dataset. Full distributions are shown in Figure 2, which plots histograms of  g for every data point in the respective corpora. (Corresponding plots for  loo are similar and the full set can be browsed via the online supplement.) We plot these separately for each class: orange ( (cid:4) ) represents instances predicted as positive, and purple ( (cid:4) ) those predicted to be negative. For SNLI, colors (cid:4) , (cid:4) and (cid:4) code for contradiction, entailment, and neutral respectively. In general, observed correlations are modest (recall: 0 indicates no correspondence, 1 implies perfect concordance) for the BiRNN encoder. The centrality of observed densities hovers around or below 0.5 in most of the corpora considered. Moreover, as per Table 2, correlation is sufficiently weak that a statistically significant correlation between attention weights and feature importance scores (both gradient and feature erasure based) cannot consistently be established across corpora. In contrast, gradients in average embedding based models show very high degree of correspondence with attention weights  on average across corpora, correlation between LOO scores and attention weights is  0.375 points higher for this encoder, compared to the BiLSTM. These results suggest that, in general, attention weights do not strongly or consistently agree with such feature importance scores in models with contextualized embeddings. This is problematic for the view of attention weights as explanatory, given the face validity of input gradient/erasure based explanations (Ross et al., 2017; Li et al., 2016). On some datasets  notably the MIMIC tasks, and to a lesser extent the QA corpora  this correlation is consistently significant but remains relatively weak. This could be attributed to increased length of documents for these datasets providing stronger signal to standard hypothesis testing methods. For reference we report correlations between gradients and LOO scores in the Appendix and online materials; these are consistently stronger than the correlation between attention weights and either feature importance score for the recurrent (BiLSTM) encoder. These exhibit, on average, a (i) 0 . 2 and (ii)  0 . 25 greater correlation with each other than BiLSTM attention and (i) LOO and (ii) gradient scores. We next consider what-if scenarios corresponding to alternative (counterfactual) attention weights. The idea is to investigate whether the prediction would have been different, had the model emphasized (attended to) different input features. More precisely, suppose   = {   t } Tt =1 are the attention weights induced for an instance, giving rise to model output  y . We then consider counterfactual distributions over y , under alternative  . We experiment with two means of constructing such distributions. First, we simply scramble the original attention weights   , re-assigning each value to an arbitrary, randomly sampled index (input feature). Second, we generate an adversarial attention distribution : this is a set of attention weights that is maximally distinct from   but that nonetheless yields an equivalent prediction (i.e., prediction within some (cid:15) of  y ). To characterize model behavior when attention weights are shuffled, we follow Algorithm 2. Figure 3 depicts the relationship between the maximum attention value in the original   and the median induced change in model output (  y med ) across instances in the respective datasets. Colors again indicate class predictions, as above. We observe that there exist many points with small  y med despite large magnitude attention weights. These are cases in which the attention weights might suggest explaining an output by a small set of features (this is how one might reasonably read a heatmap depicting the attention weights), but where scrambling the attention makes little difference to the prediction. In some cases, such as predicting ICD codes from notes using the MIMIC dataset, one can see different behavior for the respective classes. For the Diabetes task, e.g., attention behaves intuitively for at least the positive class; perturbing attention in this case causes large changes to the prediction. We again conjecture that this is due to a few tokens serving as high precision indicators for the positive class; in their absence (or when they are not attended to sufficiently), the prediction drops considerably. However, this is the exception rather than the rule. We next propose a more focused approach to counterfactual attention weights, which we will refer to as adversarial attention . The intuition is to explicitly seek out attention weights that differ as much as possible from the observed attention distribution and yet leave the prediction effectively unchanged. Such adversarial weights violate an intuitive property of explanations: shifting model attention to very different input features should yield corresponding changes in the output. Alternative attention distributions identified adversarially may then be viewed as equally plausible explanations for the same output. Operationally, realizing this objective requires specifying a value (cid:15) that defines what qualifies as a small difference in model output. Once this is specified, we aim to find k adversarial distributions {  (1) , ...,  ( k ) } , such that each  ( i ) maximizes the distance from original   but does not change the output by more than (cid:15) . In practice we simply set this to 0 . 01 for text classification and 0 . 05 for QA datasets. 6 We propose the following optimization problem to identify adversarial attention weights. maximize  (1) ,..., ( k ) f ( {  ( i ) } ki =1 ) subject to  i TVD [ y ( x ,  ( i ) ) ,  y ( x ,   )]  (cid:15) (1) Where f ( {  ( i ) } ki =1 ) is: k (cid:88) i =1 JSD [  ( i ) ,   ] + 1 k ( k  1) (cid:88) i<j JSD [  ( i ) ,  ( j ) ] (2) 6 We make the threshold slightly higher for QA because the output space is larger and thus small dimension-wise perturbations can produce comparatively large TVD. In practice we maximize a relaxed version of this objective via the Adam SGD optimizer (Kingma and Ba, 2014): f ( {  ( i ) } ki =1 ) + k (cid:80) ki =1 max(0 , TVD [ y ( x ,  ( i ) ) ,  y ( x ,   )]  (cid:15) ) . 7 Equation 1 attempts to identify a set of new attention distributions over the input that is as far as possible from the observed  (as measured by JSD) and from each other (and thus diverse), while keeping the output of the model within (cid:15) of the original prediction. We denote the output obtained under the i th adversarial attention by  y ( i ) . Note that the JS Divergence between any two categorical distributions (irrespective of length) is bounded from above by 0.69. One can view an attentive decoder as a function that maps from the space of latent input representations and attention weights over input words  T  1 to a distribution over the output space Y . Thus, for any output  y , we can define how likely each attention distribution  will generate the output as inversely proportional to TVD ( y (  ) ,  y ) . Figure 4 depicts the distributions of max JSDs realized over instances with adversarial attention weights for a subset of the datasets considered. Colors again indicate predicted class. Mass toward the upper-bound of 0.69 indicates that we are frequently able to identify maximally different attention weights that hardly budge model output. We observe that one can identify adversarial attention weights associated with high JSD for a significant number of examples. This means that is often the 7 We set  = 500 . case that quite different attention distributions over inputs would yield essentially the same (within (cid:15) output. In the case of the diabetes task, we again observe a pattern of low JSD for positive examples (where evidence is present) and high JSD for negative examples. In other words, for this task, if one perturbs the attention weights when it is inferred that the patient is diabetic, this does change the output, which is intuitively agreeable. However, this behavior again is an exception to the rule. We also consider the relationship between max attention weights (indicating strong emphasis on a particular feature) and the dissimilarity of identified adversarial attention weights, as measured via JSD, for adversaries that yield a prediction within (cid:15) of the original model output. Intuitively, one might hope that if attention weights are peaky, then counterfactual attention weights that are very different but which yield equivalent predictions 0.0 0.2 0.4 0.6 Max JS Divergence within 0.00 0.02 0.04 0.06 0.0 0.2 0.4 0.6 Max JS Divergence within [0.00,0.25) [0.25,0.50) [0.50,0.75) [0.75,1.00) M a x A tt e n t i o n Figure 5 illustrates that while there is a negative trend to this effect, it is realized only weakly. Put another way: there exist many cases (in all datasets) in which despite a high attention weight, an alternative and quite different attention configuration over inputs yields effectively the same output. In light of this, presenting a heatmap implying that a particular set of features is primarily responsible for an output would seem to be misleading. We have focused on attention mechanisms and the question of whether they afford transparency, but a number of interesting strategies unrelated to attention mechanisms have been recently proposed to provide insights into neural NLP models. These include approaches that measure feature importance based on gradient information (Ross et al., 2017; Sundararajan et al., 2017) (aligned with the gradient-based measures that we have used here), and methods based on representation erasure (Li et al., 2016), in which dimensions are removed and then the resultant change in output is recorded (similar to our experiments with removing tokens from inputs, albeit we do this at the input layer). Comparing such importance measures to attention scores may provide additional insights into the working of attention based models (Ghaeini et al., 2018). Another novel line of work in this direction involves explicitly identifying explanations of black-box predictions via a causal framework (Alvarez-Melis and Jaakkola, 2017). We also note that there has been complementary work demonstrating correlation between human attention and induced attention weights, which was relatively strong when humans agreed on an explanation (Pappas and Popescu-Belis, 2016). It would be interesting to explore if such cases present explicit high precision' signals in the text (for example, the positive label in diabetes dataset). More specific to attention mechanisms, re-cent promising work has proposed more principled attention variants designed explicitly for interpretability; these may provide greater transparency by imposing hard , sparse attention. Such instantiations explicitly select (modest) subsets of inputs to be considered when making a prediction, which are then by construction responsible for model output (Lei et al., 2016; Peters et al., 2018). Structured attention models (Kim et al., 2017) provide a generalized framework for describing and fitting attention variants with explicit probabilistic semantics. Tying attention weights to human-provided rationales is another potentially promising avenue (Bao et al., 2018). We hope our work motivates further development of these methods, resulting in attention variants that both improve predictive performance and provide insights into model predictions. We have provided evidence that correlation between intuitive feature importance measures (in-0.0 cluding gradient and feature erasure approaches) and learned attention weights is weak when using a BiRNN encoder (Section 4.1). We also established that counterfactual attention distributions  which would tell a different story about why a model made the prediction that it did  often have no effect on model output (Section 4.2). These results suggest that while attention modules consistently yield improved performance on NLP tasks, their ability to provide transparency for model predictions is (in the sense of pointing to inputs responsible for outputs) questionable. More generally, how one is meant to interpret the heatmaps' of attention weights placed over inputs that are commonly presented is unclear. These seem to suggest a story about how a model arrived at a particular disposition, but the results here indicate that the relationship between this and attention is not obvious, at least for RNN encoders. There are important limitations to this work and the conclusions we can draw from it. We have reported the (generally weak) correlation between learned attention weights and various alternative measures of feature importance, e.g., gradients. We do not imply that such alternative measures are necessarily ideal or should be considered ground truth'. While such measures do enjoy a clear intrinsic (to the model) semantics, their interpretation for non-linear neural networks can nonetheless be difficult for humans (Feng et al., 2018). Still, that attention consistently correlates poorly with multiple such measures ought to give pause to practitioners. That said, exactly how strong such correlations should' be to establish reliability as explanation is an admittedly subjective question. We note that the counterfactual attention experiments demonstrate the existence of alternative heatmaps that yield equivalent predictions; thus one cannot conclude that the model made a particular prediction because it attended over inputs in a specific way. But these adversarial weights may themselves be unlikely under the attention module parameters. Further, it may be that multiple plausible explanations exist, complicating interpretation. We would maintain that in such cases the model should highlight all plausible explanations, but one may instead view a model that provides sufficient' explanation as reasonable. An additional limitation is that we have only considered a handful of attention variants, selected to reflect common module architectures for the respective tasks included in our analysis. Alternative attention specifications may yield different conclusions; and indeed we hope this work motivates further development of principled attention mechanisms (or encoders). Finally, we have limited our evaluation to tasks with unstructured output spaces, i.e., we have not considered seq2seq tasks, which we leave for future work. However we believe interpretability is more often a consideration in, e.g., classification than in translation. We thank Zachary Lipton for insightful feedback on a preliminary version of this manuscript. This work was supported by the Army Research Office (ARO), award W911NF1810328.
abstain abstain abstain other other abstain abstain method objective method abstain abstain result method abstain abstain abstain abstain abstain abstain abstain result abstain objective objective objective abstain abstain abstain method method method abstain abstain method abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain objective method other other abstain other other other other other objective method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain objective abstain abstain result method objective method method other other