GPT2 is not fully torch.jit.trace-able
# 🐛 Bug

## Information

I'm trying to use PyTorch's tracing on a pre-trained GPT2 model and running into the following warning emitted from torch.jit.trace:
```
/opt/miniconda3/envs/py3/lib/python3.6/site-packages/transformers/modeling_gpt2.py:144: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  w = w / math.sqrt(v.size(-1))
```

Additionally, if I inspect the graph generated from the trace, I can see that the denominator in that division expression is a constant value, not one determined by the size of the tensor.

## To reproduce

The following snippet can be used to repro the problem (based on the example in the Quickstart guide)
```python
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2').eval()
example_input = torch.tensor([ tokenizer.encode("The Manhattan bridge")])
traced_model = torch.jit.trace(model, example_input)
```

## Environment info
     
- `transformers` version: 2.8.0
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (False)

see #3955 for a fix<|||||>Hi,
I have the same issue even after updating to transformers 4.1.0.  The results from the traced model are poor when compared to the original model. Following are the warnings returned when I try to create a traced version of a fine-tuned GPT2 model(using torch.jit.trace()).

`/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py:168: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  w = w / (float(v.size(-1)) ** 0.5)

/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py:173: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  mask = self.bias[:, :, ns - nd : ns, :ns]

/usr/local/lib/python3.6/dist-packages/torch/jit/_trace.py:966: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
With rtol=1e-05 and atol=1e-05, found 254695 element(s) (out of 27897075) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 5.1975250244140625e-05 (2.250950336456299 vs. 2.251002311706543), which occurred at index (0, 69, 37541).
  _module_class,`

**Environment Info**
Python: 3.6.9
PyTorch: 1.7
transformers: 4.1.0

Any suggestions would be of great help.

Thanks.<|||||>It looks like subsequent changes to `modeling_gpt2.py` have added an explicit `float` cast in that calculation, and I think that's now what is causing the trace to be incorrect.<|||||>Should we have "slow" tests on this to avoid a regression @LysandreJik @sgugger  ?<|||||>> It looks like subsequent changes to `modeling_gpt2.py` have added an explicit `float` cast in that calculation, and I think that's now what is causing the trace to be incorrect.

Thanks for your response. Is there any workaround for this? @jazzcook15 