Model2Model quickstart attention_mask dimensionality problem
## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

BERT-base-uncased

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [X] the official example scripts: (give details)
[model2model tutorial code](https://huggingface.co/transformers/quickstart.html#model2model-example)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Copy the [model2model tutorial code](https://huggingface.co/transformers/quickstart.html#model2model-example) into a new file and run it.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
Traceback (most recent call last):
  File "huggingface_m2m_example.py", line 47, in <module>
    outputs = model(question_tensor, answer_tensor, decoder_lm_labels=labels_tensor)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_encoder_decoder.py", line 234, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 997, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 819, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 433, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 394, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 334, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 257, in forward
RuntimeError: The size of tensor a (8) must match the size of tensor b (768) at non-singleton dimension 3
```

I printed out the attention masks and the attention scores [right before](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L253) and got the following:
```
question_tensor.shape:  torch.Size([1, 7])
answer_tensor.shape:  torch.Size([1, 8])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
I am a decoder
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 8, 8])
attention_masks:  torch.Size([1, 1, 7, 768])
```

It looks like this is the first time that cross attention is being called. The `is_decoder` flag is being passed as `False` in the tutorial code and we changed it to `True` in the code ourselves. The error is the same irrespective of that change.

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

The code runs as given in the tutorial.

## Environment

* OS: Linux
* Python version: 3.7.5
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): Latest master branch
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:


same issue:

Linux (ubuntu 18.04.3 LTS)

Python 3.6.9
Torch Version: 1.3.1

no GPU - regular DELL box, 

transformers installed following this part on installation guide (under python3 venv):
...
git clone https://github.com/huggingface/transformers
cd transformers
pip install .
...

Traceback (most recent call last):
  File "/home/jimihendrix/projects/transformers/albert/quickstart4_model2model.py", line 65, in <module>
    outputs = model(question_tensor, answer_tensor, decoder_lm_labels=labels_tensor)
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/transformers/modeling_encoder_decoder.py", line 234, in forward
    decoder_outputs = self.decoder(decoder_input_ids, encoder_hidden_states, **kwargs_decoder)
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/transformers/modeling_bert.py", line 986, in forward
    encoder_attention_mask=encoder_attention_mask,
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/transformers/modeling_bert.py", line 808, in forward
    encoder_attention_mask=encoder_extended_attention_mask,
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/transformers/modeling_bert.py", line 422, in forward
    hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/transformers/modeling_bert.py", line 383, in forward
    self_attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/transformers/modeling_bert.py", line 329, in forward
    hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jimihendrix/projects/transformers/venv3/lib/python3.6/site-packages/transformers/modeling_bert.py", line 253, in forward
    attention_scores = attention_scores + attention_mask
RuntimeError: The size of tensor a (8) must match the size of tensor b (768) at non-singleton dimension 3
<|||||>Indeed, I could reproduce this issue. Thanks for raising it!

My attempt at fixing it is [here](https://github.com/huggingface/transformers/pull/2452).<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
