Table 2: SpearmanR correlation of attention based importance with input gradients for 2000 samples from
the test set for the verb number prediction model.
given model given the embedding attentions as input. They differ in the assumptions they make
about how attention weights in lower layers affect
the ﬂow of information to the higher layers and
whether to compute the token attentions relative to
each other or independently.
To compute how information propagates from
the input layer to the embeddings in higher layers, it is crucial to take the residual connections
in the model into account as well as the attention
weights. In a Transformer block, both self-attention
and feed-forward networks are wrapped by residual connections, i.e., the input to these modules is
added to their output. When we only use attention
weights to approximate the ﬂow of information in
Transformers, we ignore the residual connections.
But these connections play a signiﬁcant role in
tying corresponding positions in different layers.
Hence, to compute attention rollout and attention
ﬂow, we augment the attention graph with extra
weights to represent residual connections. Given
the attention module with residual connection, we
compute values in layer l+1asVl+1=Vl+WattVl,
where Wattis the attention matrix. Thus, we have
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema