Memory is not released when moving model to CUDA
### System Info

- `transformers` version: 4.21.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): 2.8.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: GPU
- Using distributed or parallel set-up in script?: No

### Who can help?

@patil-suraj

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Steps to reproduce behavior:
1. Run Colab: https://colab.research.google.com/drive/1NWJPqwe7MOJIWd4w5LGYGaflkWXTkHGB?usp=sharing
2. Check results: 

```
model device = cuda:0
Filename: memory_leak.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     7    239.4 MiB    239.4 MiB           1   @profile
     8                                         def main():
     9    271.6 MiB     32.2 MiB           1     processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch16")
    10   1405.8 MiB   1134.2 MiB           1     model = CLIPModel.from_pretrained("openai/clip-vit-base-patch16")
    11                                           
    12   1405.8 MiB      0.0 MiB           1     device = torch.device("cuda")
    13   2644.0 MiB   1238.2 MiB           1     model = model.to(device)
    14                                         
    15   2644.5 MiB      0.5 MiB           1     print(f"model device = {model.device}")
    16   2644.5 MiB      0.0 MiB           1     gc.collect()
```

### Expected behavior

RAM should be released when a model is moved to GPU. 

This bug can be reproduced for lots of different models within the lib.
Pinging the king of memory, @ydshieh :raised_hands: <|||||>Hi @piEsposito 

As I have seen quite a few times `torch` has some of its own memory management, and related memory issues, it would be great if you can provide and example that creates a (big enough) PyTorch models (not from `transformers`) on CPU, send it to CUDA, and see if you get the memory been released.

The goal is to make sure this issue is not coming from `PyTorch`. Would you like to work on an example, please? Thanks in advance.<|||||>@ydshieh just added it to the notebook. It seems like this issue comes from PyTorch. Thanks! 