Why attention mask is -10000 but not * 0?
I am reading the code in Roberta and I found that the implementation of avoiding padding token to be calculated in self-attention is to minus its value by 10000 in function `get_extended_attention_mask`.

 I am wondering that why implement the mask by directly multiplying the value of padding tokens with zero?
Hi,

searching previous Github issues, I found this one, which might help you: #542<|||||>Thanks for the reference! While I consider that maybe implementing a softmax function that allows masking after e^x may also be an approach to implement `attention_mask`?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.