` Device to device copy is unsupporte` RuntimeError
### System Info

transformer: 4.20.1
platform: docker Ubuntu
python: 3.8.13

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am using pretrained GPT2 model to inference, and set the device to GPU in the first:

```
device = torch.device("cuda")
mode.to(device)
```

Then I got:

```
 File "/opt/conda/envs/test_environment/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/test_environment/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1048, in forward
    transformer_outputs = self.transformer(
  File "/opt/conda/envs/test_environment/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/test_environment/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 891, in forward
    outputs = block(
  File "/opt/conda/envs/test_environment/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/test_environment/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 391, in forward
    attn_outputs = self.attn(
  File "/opt/conda/envs/test_environment/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/test_environment/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 332, in forward
    attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
  File "/opt/conda/envs/test_environment/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 201, in _attn
    causal_mask = self.bias[:, :, key_length - query_length : key_length, :key_length].to(torch.bool)
RuntimeError: Device to device copy is unsupported
```

Why this errror ommitted? Thank you very much.

### Expected behavior

I didn't googled out the same problem, so I posted here. I expect this error gone.
Hi @lms-mt, thanks for reporting this issue! 

So that we can best help you, could you share the following: 
* The running environment: run `transformers-cli env` in the terminal and copy-paste the output
* A minimal code snippet to reproduce the error<|||||>cc @ArthurZucker <|||||>I am using main and I cannot reproduce this : 
```python 
>>> from transformers import GPT2Model
>>> import torch
>>> model = GPT2Model.from_pretrained("gpt2")
>>> device = torch.device("cuda")
>>> model.to("cuda")
```
works as expected<|||||>@amyeroberts Sorry to reply later. 
```
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.26.1
- Platform: Linux-5.4.0-42-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.13.4
- PyTorch version (GPU?): 2.0.0a0+gitc263bd4 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
- ```


2. Sorry I can't. because we are working on a torch backend some kind like cuda. And it should be kept private for legal.

@ArthurZucker 

Thanks for your reply. Actually, I want  to know is this message ommited from HuggingFace or torch? If I have this clue, I may resolve it myself.<|||||>@lms-mt Since `.to` method works for our supported devices e.g. `"cuda"` or `"cpu"`, then I suspect the issue is arising from the custom backend. 

As for where the error comes from, searching in both the [Hugging Face](https://github.com/search?q=org%3Ahuggingface++%22copy+is+unsupported%22&type=issues) and [PyTorch](https://github.com/search?q=org%3Apytorch+%22copy+is+unsupported%22&type=code) orgs returns no results. It's peculiar that it's raised on a line in the Hugging Face module. Is this possible that the custom backend is causing an early termination and raising the error? <|||||>> @lms-mt Since `.to` method works for our supported devices e.g. `"cuda"` or `"cpu"`, then I suspect the issue is arising from the custom backend.
> 
> As for where the error comes from, searching in both the [Hugging Face](https://github.com/search?q=org%3Ahuggingface++%22copy+is+unsupported%22&type=issues) and [PyTorch](https://github.com/search?q=org%3Apytorch+%22copy+is+unsupported%22&type=code) orgs returns no results. It's peculiar that it's raised on a line in the Hugging Face module. Is this possible that the custom backend is causing an early termination and raising the error?

@amyeroberts 
Really thanks for your reply. From what I search and your advices, I think this error may raised by the custom backend module. I will check it out. I will close this issue because this issue is not in well fromat. Have a good day.