Bart EncoderLayer masked_fill not working properly with pytorch 1.4
# 🐛 Bug


## Information

Model I am using (Bert, XLNet ...): Bart 
I'm trying to use the EncoderLayer of Bart but I realized that `attn_weights = attn_weights.masked_fill(reshaped, float("-inf"))` at line [659](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bart.py#L659) does not work when `reshaped` is an `int` or `float` tensor and throws the following error:
```
attn_weights = attn_weights.masked_fill(reshaped, float("-inf"))
RuntimeError: Expected object of scalar type Bool but got scalar type Float for argument #2 'mask' in call to _th_masked_fill_bool_
```
However it does not raise an error when I change `reshaped` type to bool but in that case it returns a tensor of `nan` values.

With @patrickvonplaten helps, we realized that it was related to a pytorch version because upgrading my torch version from 1.4 to 1.5 solved the problem

## To reproduce
```
from transformers.modeling_bart import EncoderLayer 
from transformers import BartConfig 
import torch 
hidden_states = torch.tensor(3 * [ 7 * [ 1024 * [0.4]]]) 
attn_mask = torch.ones(hidden_states.shape[:2]) 
layer = EncoderLayer(BartConfig()) 
layer(hidden_states.transpose(0, 1), attn_mask)  

```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0
- Python version: 3.6
- PyTorch version (GPU?):1.4
- Using GPU in script?: no
- Using distributed or parallel set-up in script?:no

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hello,  I still hava this problem when my pytorch were upgraded  to 1.5 .  I don't know if it's related to the python version . 
Can you give me some suggestions ？ Thank you so much!

Information
`  File "/home/ynos/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/functional.py", line 3937, in multi_head_attention_forward
    float('-inf'),
RuntimeError: Expected object of scalar type Bool but got scalar type Long for argument #2 'mask' in call to _th_masked_fill_bool_
`

Environment info:
- Python version: 3.6.2
- PyTorch version (GPU): 1.5