implementations of the models we use and algorithms we introduce publicly available at https:
//github.com/samiraabnar/attention_flow .
We start by visualizing raw attention in Figure 1a
(like Vig 2019). The example given here is correctly classiﬁed. Crucially, only in the ﬁrst couple
of layers, there are some distinctions in the attention patterns for different positions, while in higher
layers the attention weights are rather uniform. Fig-ure 2 (left) gives raw attention scores of the CLS
token over input tokens (x-axis) at different layers (y-axis), which similarly lack an interpretable
pattern.These observations reﬂect the fact that as
we go deeper into the model, the embeddings are
more contextualized and may all carry similar information. This underscores the need to track down
attention weights all the way back to the input layer
and is in line with ﬁndings of Serrano and Smith
(2019), who show that attention weights do not
necessarily correspond to the relative importance
of input tokens.
To quantify the usefulness of raw attention
weights, and the two alternatives that we consider
in the next section, besides input gradients, we
employ an input ablation method, blank-out , to estimate an importance score for each input token.
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema