attention graph, a path from node vat position k
inli, to node uat position minlj, is a series of
edges that connect these two nodes. If we look
at the weight of each edge as the proportion of
information transferred between two nodes, we
can compute how much of the information at v
is propagated to uthrough a particular path by
multiplying the weights of all edges in that path.
Since there may be more than one path between
two nodes in the attention graph, to compute the
total amount of information propagated from vtou,
we sum over all possible paths between these two
nodes. At the implementation level, to compute the
attentions from litolj, we recursively multiply the
attention weights matrices in all the layers below.
~A(li) =
A(li)~A(li1)ifi > j
A(li) ifi=j(1)
In this equation, ~Ais attention rollout, Ais raw attention and the multiplication operation is a matrix
multiplication. With this formulation, to compute
input attention we set j= 0.
Attention ﬂow In graph theory, a ﬂow network
is a directed graph with a “capacity” associated
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema