attention ) at each layer, by taking raw attentions
(i.e., embedding attention ) of that layer as well as
those from the precedent layers. These methods
are based on modelling the information ﬂow in the
network with a DAG (Directed Acyclic Graph), in
which the nodes are input tokens and hidden embeddings, edges are the attentions from the nodes
in each layer to those in the previous layer, and
the weights of the edges are the attention weights.
The ﬁrst method, attention rollout , assumes that
the identities of input tokens are linearly combined
through the layers based on the attention weights.
To adjust attention weights, it rolls out the weights
to capture the propagation of information from input tokens to intermediate hidden embeddings. The
second method, attention ﬂow , considers the attention graph as a ﬂow network. Using a maximum
ﬂow algorithm, it computes maximum ﬂow values,
from hidden embeddings (sources) to input tokens
(sinks). In both methods, we take the residual connection in the network into account to better model
the connections between input tokens and hidden
embedding. We show that compared to raw attention, the token attentions from attention rollout and
attention ﬂow have higher correlations with the importance scores obtained from input gradients as
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema