5 Conclusion
Translating embedding attentions to token attentions can provide us with better explanations about
models’ internals. Yet, we should be cautious about
our interpretation of these weights, because, we
are making many simplifying assumptions when
we approximate information ﬂow in a model with
the attention weights. Our ideas are simple and
task/architecture agnostic. In this paper, we insisted on sticking with simple ideas that only require attention weights and can be easily employed
in any task or architecture that uses self-attention.
We should note that all our analysis in this paper is
for a Transformer encoder, with no casual masking.
Since in Transformer decoder, future tokens are
masked, naturally there is more attention toward
initial tokens in the input sequence, and both attention rollout and attention ﬂow will be biased toward
these tokens. Hence, to apply these methods on a
Transformer decoder, we should ﬁrst normalize
based on the receptive ﬁeld of attention.
Following this work, we can build the attention
graph with effective attention weights (Brunner
et al., 2020) instead of raw attentions. Furthermore,
we can come up with a new method that adjusts the
attention weights using gradient-based attribution
methods (Ancona et al., 2019).
Acknowledgements
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema