model.generate() has the same speed on CPU and GPU
Hi, I find that model.generate() of BART and T5 has roughly the same running speed when running on CPU and GPU. Why doesn't GPU give faster speed? Thanks!

## Environment info
- `transformers` version: 4.1.1
- Python version: 3.6
- PyTorch version (GPU?): 1.3.1
- Using GPU in script?: yes

### Who can help
 TextGeneration: @TevenLeScao 
 Bart: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): BART and T5
## To reproduce
```python
import time
from transformers import BartTokenizer, BartForConditionalGeneration

device = 'cpu'

# change to GPU
# device = 'cuda:0'

text_to_summarize = "My friends are cool but they eat too many carbs."

tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')

inputs = tokenizer(text_to_summarize, return_tensors='pt')

inputs = inputs['input_ids'].to(device)
model = model.to(device)

start = time.time()
summary_ids = model.generate(inputs)
print("Time spent (s): ", time.time() - start)
```
## Expected behavior
I expected running on GPU should give me much faster speed. But running on GPU gave me roughly the same speed as CPU, both around 0.3s in this case.

Just realize that I used a single input... Issued closed<|||||>Thanks, your post helped me so much!
I'm using BloomModel on AWS Lambda Function but Lambda doesn't support GPU. 
So I write the code like that:
device = 'cpu'
#topic variable is already given
prompt = f' About {Topic} is what I think:  '
inputs = tokenizer(prompt, return_tensors='pt')

inputs = inputs['input_ids'].to(device)
model = model.to(device)
sample = model.generate(inputs, max_length=100, temperature=0.9, repetition_penalty = 2.0)
output = tokenizer.decode(sample[0], truncate_before_pattern=[r"\n\n^#", "^'''", "\n\n\n"])
