for attention rollout and O(d2n4)for attention
ﬂow, where dis the depth of the model, and nis
the number of tokens.
4 Analysis and Discussion
Now, we take a closer look at these three views of
attention. Figure 1 depicts raw attention, attention
rollout and attention ﬂow for a correctly classiﬁed
example across different layers. It is noteworthy
that the ﬁrst layer of attention rollout and attention
ﬂow are the same, and their only difference with
raw attention is the addition of residual connections. As we move to the higher layers, we see that
the residual connections fade away. Moreover, in
contrast to raw attention, the patterns of attention
rollout and attention ﬂow become more distinctive
in the higher layers.
Figures 2 and 3 show the weights from raw attention, attention rollout and attention ﬂow for the
CLS embedding over input tokens (x-axis) in all
6 layers (y-axis) for three examples. The ﬁrst example is the same as the one in Figure 1. The second example is “ the article on NNP large systems
<?>”. The model correctly classiﬁes this example and changing the subject of the missing verb
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema