in the next section, besides input gradients, we
employ an input ablation method, blank-out , to estimate an importance score for each input token.
Blank-out replaces each token in the input, one
by one, with UNK and measures how much it affects the predicted probability of the correct class.
We compute the Spearman’s rank correlation coefﬁcient between the attention weights of the CLS
embedding in the ﬁnal layer and the importance
scores from blank-out. As shown in the ﬁrst row
of Table 1, the correlation between raw attention
weights of the CLS token and blank-out scores is
rather low, except for the ﬁrst layer. As we can see
in Table 2 this is also the case when we compute
the correlations with input gradients.
L1 L2 L3 L4 L5 L6
Raw 0.690.27 0.100.43 -0.110.49 -0.090.52 0.200.45 0.290.39
Rollout 0.320.26 0.380.27 0.510.26 0.620.26 0.700.25 0.710.24
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema