compute values in layer l+1asVl+1=Vl+WattVl,
where Wattis the attention matrix. Thus, we have
Vl+1= (Watt+I)Vl. So, to account for residual
connections, we add an identity matrix to the attention matrix and re-normalize the weights. This
results in A= 0:5Watt+ 0:5I, where Ais the raw
attention updated by residual connections.
Furthermore, analyzing individual heads requires accounting for mixing of information between heads through a position-wise feed-forward
network in Transformer block. Using attention rollout and attention ﬂow, it is also possible to analyze
each head separately. We explain in more details
in Appendix A.1. However, in our analysis in this
paper, for simplicity, we average the attention at
each layer over all heads.
Attention rollout Attention rollout is an intuitive
way of tracking down the information propagated
from the input layer to the embeddings in the higher
layers. Given a Transformer with Llayers, we want
to compute the attention from all positions in layerlito all positions in layer lj, where j < i . In the
attention graph, a path from node vat position k
inli, to node uat position minlj, is a series of
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema