save_pretrained not working correctly when using device_map="auto" for big models in from_pretrained 
### System Info

- `transformers` version: 4.21.3
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.10.5
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.0 (True)
- Tensorflow version (GPU?): 2.9.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <True>
- Using distributed or parallel set-up in script?: <True>

### Who can help?

@patrickvonplaten

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
ML_MODEL = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto")
ML_MODEL.save_pretrained("custom_path")
```

1) Load Flan-T5-xxl using from_pretrained with device_map="auto"
In my case the model is loaded to 24 GB GPU RAM (RTX 3090) and the rest of the model ist loaded to the CPU RAM

2) Save Model to custom path using save_pretrained
In my case save_pretrained seems only to save the model information from the GPU RAM. Checking the file sizes in the custom directory only chunk 1 and 2 of the model.bin files have the expected 10GB. Chunk 3 is only partly saved and chunk 4 and 5 have only a few kB. 

### Expected behavior

When using from_pretrained without device_map="auto" the model is completely loaded into CPU RAM and also completely saved using save_pretrained. Checking the file sizes in the custom path chunk 1 to 4 have the expected 10GB and chunk5 has the expected 6GB. Same file sizes as in the Transformer Cache Directory. This behaviour is expected when using from_pretrained with device_map="auto".
Thanks for the report! Using `device_map="auto"` is only for inference, and it indeed does not work with `save_pretrained` yet, especially with offloaded weights.

We will look at adding support for this in the future!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.