we can come up with a new method that adjusts the
attention weights using gradient-based attribution
methods (Ancona et al., 2019).
Acknowledgements
We thank Mostafa Dehghani, Wilker Aziz, and the
anonymous reviewers for their valuable feedback
and comments on this work. The work presented
here was funded by the Netherlands Organization
for Scientiﬁc Research (NWO), through a Gravitation Grant 024.001.006 to the Language in Interaction Consortium.
References
Marco Ancona, Enea Ceolini, Cengiz ¨Oztireli, and
Markus Gross. 2019. Gradient-Based Attribution
Methods , pages 169–191. Springer International
Publishing.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In proceedings of
the 2015 International Conference on Learning Representations .
Gino Brunner, Yang Liu, Damian Pascual, Oliver
Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2020. On identiﬁability in transformers. In
International Conference on Learning Representations .
Hanjie Chen and Yangfeng Ji. 2019. Improving the interpretability of neural sentiment classiﬁers via data
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema