Del model does not work with device_map!=None 
### System Info

- `transformers` version: 4.29.0.dev0
- Platform: Linux-4.18.0-348.7.1.el8_5.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.7
- Huggingface_hub version: 0.13.3
- Safetensors version: 0.3.0
- PyTorch version (GPU?): 2.1.0.dev20230411+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help?

@sgugger @muellerzr

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

'del model' function does't free the GPU memory if the model has been loaded with device_map != None. 

```Python
import torch
from transformers import AutoModelForCausalLM, PreTrainedModel
import os
````

### Loading the model with device_map = None

```Python
model: PreTrainedModel = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path="EleutherAI/gpt-neo-125m",
    load_in_8bit=False,
    device_map=None,
    torch_dtype=None,
)
model = model.to("cuda")
torch.cuda.memory_allocated()
````
555601920

```Python
del model
torch.cuda.empty_cache()
torch.cuda.memory_allocated()
````
0  ✅

### Loading the model with device_map = Auto

```Python
model: PreTrainedModel = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path="EleutherAI/gpt-neo-125m",
    load_in_8bit=False,
    device_map="auto",
    torch_dtype=None,
)
torch.cuda.memory_allocated()
````
555601920

```Python
del model
torch.cuda.empty_cache()
torch.cuda.memory_allocated()
````

555077632 ❌

### Loading the model with device_map = {'': 0}

```Python
device_map = {"": int(os.environ.get("LOCAL_RANK") or 0)}

model: PreTrainedModel = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path="EleutherAI/gpt-neo-125m",
    load_in_8bit=False,
    device_map=device_map,
    torch_dtype=None,
)
torch.cuda.memory_allocated()
````
555601920

```Python
del model
torch.cuda.empty_cache()
torch.cuda.memory_allocated()
````

555077632 ❌


### Rewriting models

```Python
for x in range(1,5):
    model: PreTrainedModel = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path="EleutherAI/gpt-neo-125m",
    load_in_8bit=False,
    device_map=None,
    torch_dtype=None,
    )
    model = model.to("cuda")
    print(f"Iteration {x}: {torch.cuda.memory_allocated()}")
````
Iteration 1: 555601920
Iteration 2: 555601920
Iteration 3: 555601920
Iteration 4: 555601920


```Python
for x in range(1,5):
    model: PreTrainedModel = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path="EleutherAI/gpt-neo-125m",
    load_in_8bit=False,
    device_map="auto",
    torch_dtype=None,
    )
    print(f"Iteration {x}: {torch.cuda.memory_allocated()}")
````
Iteration 1: 554553344
Iteration 2: 1107795968
Iteration 3: 1108058112
Iteration 4: 1109368832


### Using Garbage Collector

This workaround is useful to clean the GPU memory, although it would be more appropriate to fix the delete method behavior. But for now, It can be used as a way to solve the memory leaks. 

```Python
model: PreTrainedModel = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path="EleutherAI/gpt-neo-125m",
    load_in_8bit=False,
    device_map="auto",
    torch_dtype=None,
    )
del model
torch.cuda.empty_cache()
torch.cuda.memory_allocated()
````
555077632
```Python
import gc
torch.cuda.empty_cache()
gc.collect()
torch.cuda.empty_cache()
````

```Python
torch.cuda.memory_allocated()
````
0


### Expected behavior

The model should be deleted when calling 'del model'. This bug causes multiple issues. For example: if you want to evaluate multiple model checkpoints, the model is not correctly overwritten/deleted when loading the next one, causing a memory leak that eventually results in an OOM error. 



I'm afraid using `gc.collect()` is not a workaround but the only way around this that we know of. If you find a way to have Python directly reclaim the memory without calling it, we're completely game to have it merged. I suspect it all comes down to model being initialized on the meta device where we have to re-set the parameters afterward using [this function](https://github.com/huggingface/accelerate/blob/2106e87d585ae9a245c895c568fffeaa519dfb9a/src/accelerate/utils/modeling.py#L96) but not 100% sure.

Since there is a way to avoid this by just adding a line to your cleanup code, this is not high priority for us to investigate more.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.