<?>”. The model correctly classiﬁes this example and changing the subject of the missing verb
from “article” to “article s” ﬂips the decision of the
model. The third example is “ here the NNS differ
in that the female <?>”, which is a miss-classiﬁed
example and again changing “NNS” (plural noun)
to “NNP” (singular proper noun) ﬂips the decision
of the model.
For all cases, the raw attention weights are almost uniform above layer three (discussed before).
raw attention
attention rollout
attention ﬂow
(a) “The author talked to Sara about mask book .”
raw attention
attention rollout
attention ﬂow
(b)“Mary convinced John of mask love.”
Figure 4: Bert attention maps. We look at the attention
weights from the mask embedding to the two potential
references for it, e.g. “author” and “Sara” in (a) and
“Mary” and “John” in (b). The bars, at the left, show
the relative predicted probability for the two possible
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema