computing self-attention for tokens in a sentence
## ❓ Questions & Help

Hi,

Please refer to this [issue](https://github.com/google-research/bert/issues/914).

Thanks!

To get the attention score of each heads, refer to the [documentation](https://huggingface.co/transformers/model_doc/bert.html#transformers.BertModel) : see `outputs`.

---

> how do I know which heads give the best scores ?

You can't really know. Each heads will compute some kind of attention. It can be attention for `this token refer to that token`, or it can be attention for `this token is similar to that token` or even if can be any kind of relation, even something, we human, can't interpret.  

If you want to extract some specific kind of attention, I see no other solution than manually inspect all the heads' attention and empirically choose one (or several).  

Some papers did it : [Contrastive Attention Mechanism for Abstractive Sentence Summarization](https://arxiv.org/abs/1910.13114)<|||||>Thanks for your reply.

Please refer to [here](https://github.com/huggingface/transformers/issues/2054) for more details.

Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
