use the torchscript in a gpt model is slower  than origin one.
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:2.1.1
- Platform:Linux version 4.15.0-76-generic (buildd@lcy01-amd64-029) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) 
- Python version:3.6.9
- PyTorch version (GPU?): 1.6.0+cu101
- Using GPU in script?:No
-GPU-tesla k80

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information
when i am using torchscipts to speed up the interference of my gpt2 model, I found it is slower than the origin one

traced model 0.6959998607635498
origin model 0.3259282112121582

The problem arises when using:
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] my own task : gpt2 LM

## To reproduce

Steps to reproduce the behavior:
follow the code below
https://github.com/lonelydancer/algorithm/blob/master/test.py
<!-- If you have code snippets, error messages, stack traces please provide them here as well.


     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->


## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
the traced model is faster.
Hi! TorchScript requires tracing the model beforehand, which slows down the first forward pass through the model. Could you print the timing of the iterations following the initial one?<|||||>Hi, @LysandreJik  
do you mean  in the first iteration "loaded_model(input_ids)"  will slow?
I already traced model before that.

traced_model = torch.jit.trace(model, input_ids)
torch.jit.save(traced_model, 'trace_gpt2.pt')

loaded_model = torch.jit.load('trace_gpt2.pt').to('cuda')
loaded_model.eval()

#print (loaded_model)
start = time.time()
for i in range(100):
    with torch.no_grad():
        loaded_model(input_ids)
end = time.time()
print ('traced model',(end-start))
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.