embedding. We show that compared to raw attention, the token attentions from attention rollout and
attention ﬂow have higher correlations with the importance scores obtained from input gradients as
well as blank-out , an input ablation based attribution method. Furthermore, we visualize the token
attention weights and demonstrate that they are better approximations of how input tokens contribute
to a predicted output, compared to raw attention.
It is noteworthy that the techniques we propose
in this paper, are not toward making hidden embeddings more identiﬁable, or providing better attention weights for better performance, but a new set
of attention weights that take token identity problem into consideration and can serve as a better
diagnostic tool for visualization and debugging.arXiv:2005.00928v2  [cs.LG]  31 May 2020
(a) Embedding attentions
 (b) Attention rollout
 (c) Attention ﬂow
Figure 1: Visualisation of attention weights.
Figure 2: Raw Attention maps for the CLS token at
different layers.
2 Setups and Problem Statement
In our analysis, we focus on the verb number prediction task, i.e., predicting singularity or plurality
of a verb of a sentence, when the input is the sentence up to the verb position. We use the subjectverb agreement dataset (Linzen et al., 2016). This
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema