TFRoberta output with attention_mask changes in version 2.3.0 vs 2.4.1
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): not relevant

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

```python
import tensorflow as tf
tf.get_logger().setLevel('CRITICAL')
import transformers
print(transformers.__version__)
from transformers import TFRobertaModel, RobertaConfig
from numpy.testing import assert_allclose

config = RobertaConfig()
model = TFRobertaModel(config)

input1 = tf.constant([[5, 3, 4, 8, 7, 1, 6]])
attention_mask1 = tf.constant([[1, 1, 1, 1, 1, 0, 1]])

out1, _ = model({'input_ids': input1, 'attention_mask': attention_mask1})

input2 = tf.constant([[5, 3, 4, 8, 7, 5, 6]])
attention_mask2 = tf.constant([[1, 1, 1, 1, 1, 0, 1]])

out2, _ = model({'input_ids': input2, 'attention_mask': attention_mask2})

assert_allclose(out1.numpy()[:, :5, :], out2.numpy()[:, :5, :])
```


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: I am using dummy token ids

## To reproduce

Steps to reproduce the behavior:

1. make a new virtualenv
2. install tensorflow
3. pip install transformers=2.3.0 
4. save the script in test_mask.py 
5. run python test_mask.py
6. repeat 1-5, but in 2 install the latest release: pip install transformers

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

In case of transformers==2.3.0, the test **passes**, giving the following output:

```
2.3.0
```

In case of transformers==2.4.1, the test **fails**:

```
2.4.1
Traceback (most recent call last):
  File "test_attention_mask.py", line 21, in <module>
    assert_allclose(out1.numpy()[:, :5, :], out2.numpy()[:, :5, :])
  File "/home/bartosz/.pyenv/versions/aphp-django/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/home/bartosz/.pyenv/versions/aphp-django/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 3840 / 3840 (100%)
Max absolute difference: 0.43364888
Max relative difference: 337.9916
 x: array([[[ 0.742064, -1.048889, -1.133795, ...,  1.208201, -0.110544,
         -1.556664],
        [-0.307906, -0.545374, -1.124657, ...,  0.067571, -0.857922,...
 y: array([[[ 0.718682, -0.995075, -1.105745, ...,  1.380688, -0.071943,
         -1.627201],
        [-0.390375, -0.534317, -1.113236, ...,  0.178188, -0.822041,...
```


## Expected behavior

In my understanding, the test should pass, because the only difference between inputs  `input1` and `input2` is oken with index 5 which is masked in both `attention_masks` (i.e. t, `input1[5]==1` while `input2[5]==5`). Note that I don't take the embedding for this token in the comparison of the outputs.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

     
- `transformers` version: 2.4.1
- Platform: linux
- Python version: 3.7.4
- PyTorch version (GPU?): No
- Tensorflow version (GPU?): 2.1.0 (no GPU)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
