When using GPT2，CPU  usage is high
### System Info

transformers: 4.23.1

### Who can help?

Models:
- GPT-2 @patil-suraj, @patrickvonplaten, @LysandreJik

Library
- Pipelines @Narsil 

### Reproduction

Questions: When using GPT-2 as:
```python
from transformers import pipeline
gpt2_pipe = pipeline('text-generation', model='XXX', tokenizer='gpt2')
starting_text = "a young boy"
response = gpt2_pipe(starting_text, max_length=60, num_return_sequences=1)
```
The CPU  usage keep being > 90% for a few seconds, the generation is also slow.

However, when I manually change the `transformers` package in:
https://github.com/huggingface/transformers/blob/49b77b89ea1e89a9940f2b84da1bcc0696ecb07a/src/transformers/pipelines/text_generation.py#L229

to 
```python
self.model = self.model.to('cuda')
input_ids = input_ids.to('cuda')
attention_mask = attention_mask.to('cuda')
generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
```
namely changing the variables to CUDA version, the CPU usage was much lower, and the speed is faster.

However, I could not find a good way to use cuda in `gpt2_pipline` such as `sd_pipeline` where you only need to add a `.to('cuda')`.




### Expected behavior

Can anyone give some advices? It's not an elegant way to fix the problem by changing the `transformers` package source code.
Please use the [forums](https://discuss.huggingface.co/) for such questions as we keep the issues for bugs and feature requests only.<|||||>Try

```python
gpt2_pipe = pipeline('text-generation', model='XXX', tokenizer='gpt2', device=0) # or `cuda:0`
```