attention_mask added, not multiplied ... is this correct?
https://github.com/huggingface/transformers/blob/master/transformers/modeling_bert.py#L233

This line adds (+ operator) the attention mask. I wonder whether this is correct, as I would have very much expected the mask to be multiplied.
Ping. :) To me this still looks like the code actually fails to apply the attention mask and also the parts of the sequence intended to be masked are accessible. The line is now https://github.com/huggingface/transformers/blob/master/transformers/modeling_bert.py#L237<|||||>Hi, yes this is correct. Inside the `BertModel` forward method, the `attention_mask` is set to `0` for the tokens which should be attended (no modification) and `-10000` for the tokens which must be ignored, resulting in nullification of their attention scores.

You can read the relevant source code [here](https://github.com/huggingface/transformers/blob/master/transformers/modeling_bert.py#L683).<|||||>I see. Thank you very much for the informative answer!<|||||>Thank you for the explanation. I was thinking the same thing when reading the code.

In that case, shouldn't the `attention_mark` input for BertEncoder, BertLayer, ...  be renamed to `extended_attention_mask` or `scaled_attention_mark`? Because those inner modules do expect the scaled (and reshaped?) mark, not the user input attention_mark.

Just a suggestion.
