International Conference on Learning Representations .
Hanjie Chen and Yangfeng Ji. 2019. Improving the interpretability of neural sentiment classiﬁers via data
augmentation. arXiv preprint arXiv:1909.04225 .
Kevin Clark, Urvashi Khandelwal, Omer Levy, and
Christopher D. Manning. 2019. What does BERT
look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP:
Analyzing and Interpreting Neural Networks for
NLP, pages 276–286, Florence, Italy. Association
for Computational Linguistics.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim,
Adam Pearce, Fernanda Vi ´egas, and Martin Wattenberg. 2019. Visualizing and measuring the geometry
of bert. arXiv preprint arXiv:1906.02715 .
Thomas H. Cormen, Charles E. Leiserson, Ronald L.
Rivest, and Clifford Stein. 2009. Introduction to
Algorithms, Third Edition , 3rd edition. The MIT
Press.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals,
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.
Samira Abnar Willem Zuidema