more focused compared to attention ﬂow weights,
which is sensible for the third example but not as
much for the second one.
L1 L3 L5 L6
Raw 0.120.21 0.090.21 0.080.20 0.090.21
Rollout 0.110.19 0.120.21 0.130.21 0.130.20
Flow 0.110.19 0.110.21 0.120.22 0.140.21
Table 3: SpearmanR correlation of attention based importance with input gradients for 100 samples from the
test set for the DistillBERT model ﬁne tuned on SST-2.
Furthermore, as shown in Table 1 and 2 both
attention rollout and attention ﬂow, are better
correlated with blank-out scores and input gradients compared to raw attention, but attention ﬂow
weights are more reliable than attention rollout.
The difference between these two methods is rooted
in their different views of attention weights. Attention ﬂow views them as capacities, and at every
step of the algorithm, it uses as much of the capacity as possible. Hence, attention ﬂow computes the
maximum possibility of token identities to propagate to the higher layers. Whereas attention rollout
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema