glue benchmark (Wang et al., 2018). Even though
for this model, all three methods have very low
correlation with the input gradients, we can still see
that attention rollout and attention ﬂow are slightly
better than raw attention.
Furthermore, in Figure 4, we show an example
of applying these methods to a pre-trained Bert
to see how it resolves the pronouns in a sentence.
What we do here is to feed the model with a sentence, masking a pronoun. Next, we look at the
prediction of the model for the masked word and
compare the probabilities assigned to “her” and
“his”. Then we look at raw attention, attention rollout and attention ﬂow weights of the embeddings
for the masked pronoun at all the layers. In the
ﬁrst example, in Figure 4a, attention rollout and
attention ﬂow are consistent with each other and
the prediction of the model. Whereas, the ﬁnal
layer of raw attention does not seem to be consistent with the prediction of the models, and it varies
a lot across different layers. In the second example, in Figure 4b, only attention ﬂow weights areconsistent with the prediction of the model.
5 Conclusion
Translating embedding attentions to token attentions can provide us with better explanations about
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema