[DistilBert] Flaky tests
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Linux-5.3.0-61-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyTorch version (GPU?): 1.6.0+cpu (False)
- Tensorflow version (GPU?): 2.1.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Reproduce Error:

```python 
#!/usr/bin/env python3
import torch
from transformers import DistilBertModel, DistilBertConfig

input_ids = torch.tensor([[55, 40, 88, 37, 12,  6, 20],
        [33, 87, 56,  6, 34, 92,  2],
        [ 4, 25, 95, 19,  9, 14, 80],
        [96, 45, 71, 10, 78, 33, 68],
        [72, 40, 59, 90,  5, 78, 44],
        [36, 15, 11, 18, 74, 40, 30],
        [84, 25,  5, 61, 18, 77, 35],
        [70, 87,  9, 42, 24, 65, 11],
        [28,  0, 28, 45, 92, 83, 96],
        [75, 41, 69, 61, 83, 31, 81],
        [94, 93, 79, 48, 24, 17,  9],
        [97,  5, 38, 94, 75,  8, 59],
        [31, 71, 87, 39, 97, 10, 22]])

attention_mask = torch.tensor([[1, 1, 1, 1, 0, 1, 0],
        [0, 0, 0, 0, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1],
        [0, 1, 1, 1, 0, 0, 1],
        [1, 1, 1, 0, 1, 1, 0],
        [1, 1, 1, 1, 1, 1, 1],
        [0, 0, 1, 0, 0, 1, 1],
        [1, 0, 1, 0, 0, 0, 1],
        [0, 1, 0, 1, 1, 1, 0],
        [0, 1, 1, 0, 0, 0, 0],
        [0, 1, 0, 1, 1, 0, 1],
        [0, 1, 1, 1, 1, 1, 1],
        [0, 1, 0, 1, 0, 0, 0]])


distil_bert_config = {
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 32,
  "dropout": 0.1,
  "hidden_act": "gelu",
  "hidden_dim": 37,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 4,
  "n_layers": 5,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "return_dict": True,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": False,
  "vocab_size": 99
}

config = DistilBertConfig(**distil_bert_config)
torch.manual_seed(0)
model = DistilBertModel(config).eval()

last_hidden_state = model(input_ids, attention_mask=attention_mask)[0]

if torch.isnan(last_hidden_state).any().item():
    print("Error with DistilBert")
```

This code example allows yields nan values.
Will investigate now @sgugger @VictorSanh . My first guess is that it because of "inf" values because of masking because this error does not happen when the attention mask is not passed to forward. 