Methods in Natural Language Processing: System
Demonstrations , pages 121–126.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg.
2016. Assessing the ability of LSTMs to learn
syntax-sensitive dependencies. Transactions of the
Association for Computational Linguistics , 4:521–
535.
Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. 2019. Learning
to deceive with attention-based explanations. arXiv
preprint arXiv:1909.07913 .Alec Radford, Jeffrey Wu, Rewon Child, David Luan,
Dario Amodei, and Ilya Sutskever. 2019. Language
models are unsupervised multitask learners. OpenAI
Blog , 1(8).
Tim Rockt ¨aschel, Edward Grefenstette, Karl Moritz
Hermann, Tomas Kocisky, and Phil Blunsom. 2016.
Reasoning about entailment with neural attention.
InInternational Conference on Learning Representations (ICLR) .
Victor Sanh, Lysandre Debut, Julien Chaumond, and
Thomas Wolf. 2019. Distilbert, a distilled version of
bert: smaller, faster, cheaper and lighter.
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema