It’s an old issue but I think I found something related to this issue recently.
When I looked into my site-packages/torch/nn/functional.py, which I assume to be the path to MultiheadAttention function, I found that the attention scores are not divided by, as suggested in the Transformer paper, the square root of attention dimension. As the reason to this division in the paper suggest, this may saturate softmax function.