Loading LLaMA model does not use GPU memory neither offload folder
### System Info

- `transformers` version: 4.32.0.dev0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): Tensorflow version (GPU?): 2.13.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes (**this is part of the issue**)
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

![2023-07-30 105328](https://github.com/huggingface/transformers/assets/31779190/11d18954-ad61-43e0-a12e-42b95ebd75b1)

code used:
```
from pynvml import *
nvmlInit()
h = nvmlDeviceGetHandleByIndex(0)
info = nvmlDeviceGetMemoryInfo(h)

print(f'total    : {info.total}')
print(f'free     : {info.free}')
print(f'used     : {info.used}')

from transformers import LlamaForCausalLM, LlamaTokenizer

model_id="./converted-llama-2-7b-chat"

tokenizer = LlamaTokenizer.from_pretrained(model_id)
model =LlamaForCausalLM.from_pretrained(model_id, device_map="auto", offload_folder="offload")
```

The memory tops and script finish with **killed** message without usage of offload folder and usage of 618 MiB of GPU RAM only

### Expected behavior

- Usage of available GPU RAM
- Usage of offload folder

before finishing with **killed**  message
cc @sgugger <|||||>Hi @not4fame , can you print the `device_map` of the model by printing `model.hf_device.map` ? What do you mean that it is not using the `offloaded folder` ? To me, if you were able to load your model without error, it means that you are indeed using disk offload as your GPU RAM + RAM < model size. As for the gpu, we need to leave some space to bring back the layers that were offloaded to the cpu to the gpu during inference. To make better use of your gpus, you should probably load the model in fp16. Check out this [colab](https://colab.research.google.com/drive/11HJsgGJl8eK57FEPVmHmfnxzbfod55yM?usp=sharing) where load the model in fp16. <|||||>Also, you might need to use `offload_state_dict=True` to avoid getting out of CPU RAM while loading your model.<|||||>Thank you @sgugger `offload_state_dict=True` flag was exactly what was missing. Now my script is properly using offload folder.

![2023-08-01 070505](https://github.com/huggingface/transformers/assets/31779190/cdad57f4-928b-403d-960d-b256c6b58e9b)

thank you @SunMarc  for your comments, I wasn't able to dump the device_map because the model didn't load, but when trying to access this configuration, after some initial research, I managed to find a way to have even more control when loading the model which resulted in better performance
```
from pynvml import *
nvmlInit()
h = nvmlDeviceGetHandleByIndex(0)
info = nvmlDeviceGetMemoryInfo(h)

print(f'total    : {info.total}')
print(f'free     : {info.free}')
print(f'used     : {info.used}')

from transformers import LlamaForCausalLM, LlamaTokenizer, AutoConfig, AutoModelForCausalLM
import accelerate
import json

model_id="./converted-llama-2-7b-chat"

config = AutoConfig.from_pretrained(model_id)
with accelerate.init_empty_weights():
    dummy_model = AutoModelForCausalLM.from_config(config)
device_map = accelerate.infer_auto_device_map(dummy_model, max_memory={0: "4GiB", "cpu": "10GiB"})

tokenizer = LlamaTokenizer.from_pretrained(model_id)
model =LlamaForCausalLM.from_pretrained(
   model_id,
   device_map=device_map,
   load_in_8bit=True,
   llm_int8_enable_fp32_cpu_offload=True,
   offload_folder="offload",
   offload_state_dict=True)
```