Attention values occasionally exceed 1 in BertModel
```Python
outputs = self.model(x, attention_mask = x_mask)  # Models outputs are now tuples
print(outputs[2][-1].max())
print((outputs[2][-1]>1).sum().item()) # Number of attention values > 1
print((outputs[2][-1]>-1).sum().item()) # Total number of attention values
```
```
tensor(1.0750, device='cuda:7', grad_fn=<MaxBackward1>)
1545
480000
```
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@thomwolf Did someone looking into this issue?<|||||>No and I'm afraid we don't really have the bandwidth for that at the moment.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I think this is related to a feature of Dropout applied to the attention values. Dropout will scale up the values that are not zeroed out which causes the problem you described. See this [pytorch/issues/5752](https://github.com/pytorch/pytorch/issues/5752). 
Setting the model to eval mode should produce normal attention values.