`import decord` crashes Python kernel/process when moving X-CLIP or other video classification models to CUDA GPU
### System Info

- `transformers` version: 4.25.1
- Platform: Linux-6.0.12-76060006-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sgugger for the docs and @NielsRogge for X-CLIP specifically.

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I was trying to run inference on GPU with X-CLIP on my own dataset. To do so I followed the [example code](https://huggingface.co/docs/transformers/main/en/model_doc/xclip#transformers.XCLIPModel.forward.example) in the docs, which uses decord as library to load videos into memory. I tested it and it ran perfectly, _but_ I noticed the model was on CPU.  
A simple `model.to(torch.device("cuda"))` ought to do the trick, right? Well no, here is the rabbit hole I went down: https://github.com/huggingface/transformers/issues/21054.

My conclusion is that a simple `import decord` before trying to load the model into GPU memory is enough to make the `python` process crash, be it in the terminal or a Jupyter kernel.  
This is both with the latest decord version from PyPI (which can only run on CPU) and with the latest decord version compiled from source with CUDA enabled.

To fix this, I used the [code](https://colab.research.google.com/gist/nateraw/c327cb6ff6b074e6ddc8068d19c0367d/pyav-io.ipynb#scrollTo=fzGRpWaUqnTL) generously provided by @nateraw built on pyAV, while discussing how to integrate videos in the datasets library (https://github.com/huggingface/datasets/issues/5225).

### Expected behavior

The model should be transferable to the GPU without issue.  
To instruct people on how to do so, the docs should be updated to make use of pyAV instead of decord to avoid sending other users into hours if not days of debugging.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Unstale<|||||>cc @amyeroberts 