fp16 is not compatible with the current activation code when pytorch is less than 1.4.0
[gelu = getattr(F, "gelu", _gelu_python)](https://github.com/huggingface/transformers/blob/53ce3854a16ad2a715bc6ac8af3e30c18b5a1d11/src/transformers/activations.py#L21)

should be changed to:

```python
if torch.__version__ < '1.4.0':
    gelu = _gelu_python
else:
    gelu = getattr(F, "gelu", _gelu_python)
```


This is a duplicate of https://github.com/huggingface/transformers/issues/2940