Padding changes model outputs (even with attention_mask)
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Consider the following example code which gets a BERT embedding, and compares the BERT embedding on the same input with padding + masking.
```
bertModel=BertModel.from_pretrained('bert-base-uncased')
bertModel=bertModel.eval()
print('token id:'+str(id))
print('token types:'+str(typeout))

bert_out = torch.mean(bertModel(id, token_type_ids = typeout, attention_mask = mask)[0],1)

add_pad = lambda x: torch.cat((x, torch.zeros(1, 10, dtype=x.dtype)),1)
print('mask:'+str(mask))
print('padded id:'+str(add_pad(id)))
print('padded type:'+str(add_pad(typeout)))
print('padded mask:'+str(add_pad(mask)))
bert_out_2 = torch.mean(bertModel(add_pad(id), token_type_ids = add_pad(typeout), attention_mask = add_pad(mask))[0],1)

print(bert_out[0,0:10])
print(bert_out_2[0,0:10])
```
The output here is 
```
token id:tensor([[ 101, 5292, 3270,  102, 8638, 2060,  102]])
token types:tensor([[0, 0, 0, 0, 1, 1, 1]])
mask:tensor([[1., 1., 1., 1., 1., 1., 1.]])
padded id:tensor([[ 101, 5292, 3270,  102, 8638, 2060,  102,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0]])
padded type:tensor([[0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
padded mask:tensor([[1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
tensor([ 0.2600,  0.1568, -0.2576, -0.0110,  0.1021, -0.1336,  0.2546, -0.5071,
         0.0462, -0.2283], grad_fn=<SliceBackward>)
tensor([ 0.0996,  0.0061, -0.3331, -0.0237, -0.1110, -0.0050,  0.2755, -0.3335,
        -0.0565, -0.2542], grad_fn=<SliceBackward>)
```
The two tensors which were generated by the same input but have different padding have drastically different embeddings.


## Expected behavior

The last two lines of the output should be the same
## Environment info
     
- `transformers` version: 2.1.1.
- Platform: osx
- Python version: 3.7.3
- PyTorch version (GPU?): 1.2.0
- Tensorflow version (GPU?): N/A
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

I tried to replicate the input as close as I could given the output you gave:

```py
from transformers import BertModel
import torch

token_id = torch.tensor([[ 101, 5292, 3270,  102, 8638, 2060,  102]])
token_types = torch.tensor([[0, 0, 0, 0, 1, 1, 1]])
mask = torch.tensor([[1., 1., 1., 1., 1., 1., 1.]])

padded_id = torch.tensor([[ 101, 5292, 3270,  102, 8638, 2060,  102,    0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
padded_type = torch.tensor([[0, 0, 0, 0, 1, 1, 1,                       0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
padded_mask = torch.tensor([[1., 1., 1., 1., 1., 1., 1.,                0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

bertModel = BertModel.from_pretrained('bert-base-uncased')
bertModel = bertModel.eval()

output = bertModel(token_id, token_type_ids=token_types, attention_mask=mask)
padded_output = bertModel(padded_id, token_type_ids=padded_type, attention_mask=padded_mask)
```

I then print the maximum difference between the output of the model with the non-padded input and the output with the padded input:

```py
print(torch.max(output[0] - padded_output[0][:, :7]))
print(torch.max(output[1] - padded_output[1]))
```

Which outputs the following (negligible) difference:

```py
tensor(3.6359e-06, grad_fn=<MaxBackward1>)
tensor(5.9605e-07, grad_fn=<MaxBackward1>)
```

Would it be possible for you to give a completely reproducible script so that I may see where the issue lies?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
