What is the meaning of Attention Mask
Hi, I noticed that there is something called `Attention Mask` in the model.

In the annotation of class `BertForQuestionAnswering`, 

```python
`attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices
            selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
            input sequence length in the current batch. It's the mask that we typically use for attention when
            a batch has varying length sentences.
```

And its usage is in class `BertSelfAttention`, function `forward`,

```python
# Apply the attention mask is (precomputed for all layers in BertModel forward() function)
attention_scores = attention_scores + attention_mask
```

It seems the attention_mask is used to add 1 to the scores for positions that is taken up by real tokens, and add 0 to the positions outside current sequence. 

Then, why not set the scores to `-inf` where the positions are outside the current sequence. Then pass the scores to a softmax layer, those score will become 0 as we want.
Yes, this conversion is done inside the model, see this line: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L626
(we don't use infinity but a large value that works also when the model is used in half precision mode)<|||||>Thanks for your answer. Well, I still have a little problem understanding what you mean in the last sentence:

> (we don't use infinity but a large value that works also when the model is used in half precision mode)

I make a simple experiment:

```python
>>> import torch
>>> import numpy as np
>>> inf = np.array(-np.inf)
>>> inf
array(-inf)
>>> inf = torch.from_numpy(inf)
>>> inf
tensor(-inf, dtype=torch.float64)
>>> inf.half()
tensor(-inf, dtype=torch.float16)
```

It seems `-inf` works well.
In another experiment, I tried the following: 

```python
>>> scores = torch.FloatTensor([1, 2, 3, 4, 4])
>>> mask = torch.ByteTensor([1, 1, 1, 0, 0])
>>> scores.masked_fill_(mask == 0, -np.inf)
tensor([1., 2., 3., -inf, -inf])
>>> scores.half()
tensor([1., 2., 3., -inf, -inf], dtype=torch.float16)
```
It seems that both 2 experiments works well, so I don't get what is the problem to use `-inf` in half precision mode

Thank you<|||||>attention mask is -10000.0 for positins that we do not want attention for

it is set to -10000.0 in 

        # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
        # masked positions, this operation will create a tensor which is 0.0 for
        # positions we want to attend and -10000.0 for masked positions.
        # Since we are adding it to the raw scores before the softmax, this is
        # effectively the same as removing these entirely.
        
in [get_extended_attention_mask](https://github.com/huggingface/transformers/blob/e95d433d77727a9babadf008dd621a2326d37303/src/transformers/modeling_utils.py#L700)