“Mary” and “John” in (b). The bars, at the left, show
the relative predicted probability for the two possible
pronouns, “his” and “her”.
In the case of the correctly classiﬁed example, we
observe that both attention rollout and attention
ﬂow assign relatively high weights to both the subject of the verb, “article’ and the attractor, “systems”. For the miss-classiﬁed example, both attention rollout and attention ﬂow assign relatively
high scores to the “NNS” token which is not the
subject of the verb. This can explain the wrong
prediction of the model.
The main difference between attention rollout
and attention ﬂow is that attention ﬂow weights are
amortized among the set of most attended tokens,
as expected. Attention ﬂow can indicate a set of
input tokens that are important for the ﬁnal decision.
Thus we do not get sharp distinctions among them.
On the other hand, attention rollout weights are
more focused compared to attention ﬂow weights,
which is sensible for the third example but not as
much for the second one.
L1 L3 L5 L6
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema