Find dot product of query and key vectors 
Hi,

I am following [this popular article](http://jalammar.github.io/illustrated-transformer/) to understand the Transformers. Alongside this, I am using [huggingface transformers](https://huggingface.co/transformers/model_doc/bert.html#bertmodel) to get the attention scores.

On running the following code:
`from transformers import BertTokenizer, BertModel, BertConfig, BertForTokenClassification
import torch

config = BertConfig.from_pretrained('bert-base-uncased', output_hidden_states=True, output_attentions=True)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased', config=config)

input_ids = torch.tensor(tokenizer.encode("Hello my dog is cute", add_special_tokens=False)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)

print(len(outputs))

last_hidden_states, pooler_outputs, hidden_states, attentions = outputs  # The last hidden-state is the first element of the output tuple

print(attentions)`

I get the weighted sum attention matrix of size 5x5. I am actually trying to find the softmax values like 0.88 and 0.12. I was wondering if there is any way I can obtain the dot-product scores.

![image](https://user-images.githubusercontent.com/22553367/70174075-394bbf80-16a2-11ea-972b-4aeb539dbd7e.png)

Thanks!
I found [this](https://huggingface.co/transformers/_modules/transformers/modeling_bert.html) code which has transpose_for_scores but I am not sure how this can be used with the above code.<|||||>Yes, the `attentions` outputs of the model are the softmax values.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
