device_map="auto" -> uninitialized parameters
### System Info

- `transformers` version: 4.31.0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)


### Who can help?

@ArthurZucker @younesbelkada
Maybe also @sgugger because this is a general use-case about PyTorch models

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am encountering an issue that worries me slightly. When I load a model with `device_map`, everything goes fine - no warnings.

```python
from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("BramVanroy/flan-t5-small-amr-en")
```

Howver, when I do use the device_map, I get the warning that some weights are not initialized

```python
from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("BramVanroy/flan-t5-small-amr-en", device_map="auto")
```

Result:
> Some weights of T5ForConditionalGeneration were not initialized from the model checkpoint at BramVanroy/flan-t5-small-amr-en and are newly initialized: ['decoder.embed_tokens.weight', 'encoder.embed_tokens.weight']
> You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

However, I am wondering whether this isn't a false positive because the model performance seems the same with/without. My model repo contains both safetensors and the PyTorch *.bin, if that has something to do with it?

### Expected behavior

Either a warning in both or no warning in either.
I think this should have been fixed by #25101 Could you try again with a source install?
(Yes it is a false positive, just tied weights where the copies are not present in the state dict.)<|||||>Awesome, that works. Was afraid that I was messing something up with converting to safetensors. Glad that that is not the case.

Thanks for the prompt response! @sgugger 