of a verb of a sentence, when the input is the sentence up to the verb position. We use the subjectverb agreement dataset (Linzen et al., 2016). This
task and dataset are convenient choices, as they offer a clear hypothesis about what part of the input
is essential to get the right solution. For instance,
given “ the key to the cabinets ” as the input, we
know that attending to “key” helps the model predict singular as output while attending to “cabinets”
(anagreement attractor , with the opposite number)
is unhelpful.
We train a Transformer encoder, with GPT2 Transformer blocks as described in (Radford
et al., 2019; Wolf et al., 2019) (without masking).
The model has 6 layers, and 8 heads, with hidden/embedding size of 128. Similar to Bert (Devlin et al., 2019) we add a CLS token and use
its embedding in the ﬁnal layer as the input to
the classiﬁer. The accuracy of the model on the
subject-verb agreement task is 0:96. To facilitate
replication of our experiments we will make the
implementations of the models we use and algorithms we introduce publicly available at https:
//github.com/samiraabnar/attention_flow .
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema