2017) has become the key building block of neural sequence processing models, and visualizing
attention weights is the easiest and most popular
approach to interpret a model’s decisions and to
gain insights about its internals (Vaswani et al.,
2017; Xu et al., 2015; Wang et al., 2016; Lee et al.,
2017; Dehghani et al., 2019; Rockt ¨aschel et al.,
2016; Chen and Ji, 2019; Coenen et al., 2019; Clark
et al., 2019). Although it is wrong to equate attention with explanation (Pruthi et al., 2019; Jain and
Wallace, 2019), it can offer plausible and meaningful interpretations (Wiegreffe and Pinter, 2019;
Vashishth et al., 2019; Vig, 2019). In this paper,
we focus on problems arising when we move to the
higher layers of a model, due to lack of token identiﬁability of the embeddings in higher layers (Brunner et al., 2020).We propose two simple but effective methods to
compute attention scores to input tokens (i.e., token
attention ) at each layer, by taking raw attentions
(i.e., embedding attention ) of that layer as well as
those from the precedent layers. These methods
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema