maximum possibility of token identities to propagate to the higher layers. Whereas attention rollout
views them as proportion factors and at every step,
it allows token identities to be propagated to higher
layers exactly based on this proportion factors. This
makes attention rollout stricter than attention ﬂow,
and so we see that attention rollout provides us with
more focused attention patterns. However, since
we are making many simplifying assumptions, the
strictness of attention rollout does not lead to more
accurate results, and the relaxation of attention ﬂow
seems to be a useful property.
At last, to illustrate the application of attention ﬂow and attention rollout on different tasks
and different models, we examine them on two
pretrained BERT models. We use the models
available at https://github.com/huggingface/
transformers .
Table 3 shows the correlation of the importance
score obtained from raw attention, attention rollout
and attention ﬂow from a DistillBERT (Sanh et al.,
2019) model ﬁne-tuned to solve “SST-2” (Socher
et al., 2013), the sentiment analysis task from the
glue benchmark (Wang et al., 2018). Even though
for this model, all three methods have very low
correlation with the input gradients, we can still see
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema