Is the attention_mask in BertSelfAttention applied correctly?
https://github.com/huggingface/transformers/blob/69511cdcaec8c1c7f0d7f378964eca0ce74ed5a8/src/transformers/models/bert/modeling_bert.py#L325-L327

Relevant Models:
- BERT: @LysandreJik 

I was just working on adjusting Bert to my custom architecture, and when editing the BertSelfAttention module, I have noticed a very strange couple of lines (see the linked code). Shouldn't the masking be applied multiplicatively instead of additively? :thinking: 

I'm happy to be proven wrong and learn a new thing, but it seemed worth bringing up.
These issues may be of help: https://github.com/huggingface/transformers/issues/13555 https://github.com/huggingface/transformers/issues/542

Apparently it would still help quite a bit to have comments explaining what's happening - would you like to try your hand at it?<|||||>Oooh, yeah, didn't realize it was right before softmax, I had to solve this issue before...

I found the code that converts the binary mask into the log-scale mask as well.

Sure, I will update it and post a pr during the weekend. :)<|||||>hmm this piece of code is also present in multiple models I'm sure, all of the models using masked attention could use documenting, correct? @LysandreJik <|||||>Yes, if you could add 1-2 lines of doc to the models concerned, that would be of great help! Thanks, @avolny!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.