network, a maximum ﬂow algorithm ﬁnds a ﬂow
which has the maximum possible value between s
andt(Cormen et al., 2009).
Treating the attention graph as a ﬂow network,
where the capacities of the edges are attention
weights, using any maximum ﬂow algorithm, we
can compute the maximum attention ﬂow from any
node in any of the layers to any of the input nodes.
We can use this maximum-ﬂow-value as an approximation of the attention to input nodes. In attention
ﬂow, the weight of a single path is the minimum
value of the weights of the edges in the path, instead of the product of the weights. Besides, we
attention rollout
 attention ﬂow
Figure 3: Attention maps for the CLS token
.
can not compute the attention for node sto node
tby adding up the weights of all paths between
these two nodes, since there might be an overlap
between the paths and this might result in overﬂow
in the overlapping edges.
It is noteworthy that both of the proposed methods can be computed in polynomial time. O(dn2)
for attention rollout and O(d2n4)for attention
ﬂow, where dis the depth of the model, and nis
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema