X-CLIP and other video classification models can't be loaded into CUDA GPU for inference without crashing the kernel/process
### System Info

Originally:
- `transformers` version: 4.25.1 (also tried 4.26.0-dev directly from the GitHub main branch)
- Platform: Linux-6.0.12-76060006-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

Then, given [this comment](https://github.com/microsoft/VideoX/issues/57#issuecomment-1283627674) in the X-CLIP issues, I also tried:
- `transformers` version: 4.25.1
- Platform: Linux-6.0.12-76060006-generic-x86_64-with-glibc2.35
- Python version: 3.8.16
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.8.0+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@NielsRogge tagging you since you've added the code for X-CLIP to the library and also commented in the X-CLIP issue I've mentioned above.

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. I have first copied [the example code](https://huggingface.co/docs/transformers/main/en/model_doc/xclip#transformers.XCLIPModel.forward.example) provided in the library documentation, which worked. 
2. Then I've extended my notebook to process data from my (currently) private dataset, but still following exactly the example code. This is where I've noticed that the inference took a few seconds, so...
3. I have compiled [decord](https://github.com/dmlc/decord) from source, which allowed me to run the data processing on the GPU. This worked, but it didn't provide any performance improvement, so I reverted to the PyPI version.
4. I tried manually moving the model to the GPU, with `model.to("cuda")`, `model.to("cuda:0")`, `model.to(torch.device("cuda"))` and `model.cuda()`. All of these make the Jupyter Lab kernel crash with no error in the logs. If reloaded, the model still works, but only runs on CPU.
5. I also tried replacing `XClipModel` with other video classification models, such as [`TimesformerForVideoClassification`](https://huggingface.co/docs/transformers/main/en/model_doc/timesformer#transformers.TimesformerForVideoClassification). Since this model is not included in the stable release yet, I uninstalled transformers v4.25.1 and installed the current main branch (v4.26.0-dev). This still only ran on CPU and refused to work on GPU.
6. I have then found [this comment](https://github.com/microsoft/VideoX/issues/57#issuecomment-1283627674) about my exact problem in the microsoft/VideoX issues, saying they solved it by downgrading to PyTorch 1.8.0, which I did (from 1.13.0) after also downgrading Python (from 3.10 to 3.8 due to PyTorch compatibility). With this change, instantiating the model made the kernel crash immediately. My guess is that between PyTorch 1.8.0 and 1.13.0 a fallback to the CPU if the model couldn't be loaded into GPU was introduced.

Other details:
- Linux distro: Pop!_OS 22.04
- CPU: Ryzen 5 5600X
- GPU: NVIDIA RTX 3090
- RAM: 16GB (even though limited, the model which I'm trying to load (microsoft/xclip-base-patch16-zero-shot) should fit with no problem)
- NVIDIA driver 525.60.11
- CUDA 11.2 (installed with the `system76-cuda-latest` metapackage) -- even though `nvidia-smi` reports CUDA 12.0, could this be an issue?

### Expected behavior

The model should be loaded into the GPU automatically, like other models that currently work flawlessly for me such as BART. At least, manually moving the model to the GPU should work without segfaulting.
After trying to revert the driver back from 525 to 515 and installing CUDA with other methods, such as by specifying `cuda_toolkit=11.7` in the `conda` installation (I originally just used `venv`), I've found out that my original configuration (under "Other details" above) worked if I did not import decord.  

The issue is given by simply importing decord (`import decord` is sufficient) before trying to move the model to the GPU. Unfortunately, since decord is written in C at its core, the Python process simply segfaults without error.  
I'm now using pyAV and everything works as expected. I'm closing this issue and opening another one to ask for updated docs without decord, this has cost me a lot of debugging time for something simple but undocumented, so I hope to save that time to other users.<|||||>Hi @e-caste,

Thanks a lot for investigating. I'm not able to reproduce this in [Google Colab](https://colab.research.google.com/drive/1SMc0zW_zfp8j-iiasUh3CeJMY2i1NlPu?usp=sharing), which at the moment has PyTorch 1.13. I'm using the main branch of Transformers. The model seems correctly placed on the GPU (which I confirmed by running `nvidia-smi` and seeing whether the memory is occupied).

I'll ping @nateraw here as he has been looking into several video decoding libraries, we should of course take one that works as intended. From [this thread](https://github.com/huggingface/datasets/issues/5225), we're currently in favor of using PyAV.<|||||>@NielsRogge I'm not sure if literally the `import decord` line is enough for the kernel to crash (I've tested a lot of things and I can't remember), but I'm sure that the import line in the docs (`from decord import VideoReader, cpu` -- I had it before `import torch` and `from transformers import XCLIPProcessor, XCLIPModel`) made it crash.

