Model isn't loaded with the right type with AutoModel with torch_dtype="auto" 
### System Info

- `transformers` version: 4.30.0.dev0

### Who can help?

@stas00 
(as the relevant change was made in https://github.com/huggingface/transformers/pull/21524)

### Reproduction

```python
from transformers import AutoModelForSeq2SeqLM, T5ForConditionalGeneration

auto_model = AutoModelForSeq2SeqLM.from_pretrained("ybelkada/flan-t5-xl-sharded-bf16", torch_dtype="auto")
print(auto_model.dtype)  # torch.float32

t5_model = T5ForConditionalGeneration.from_pretrained("ybelkada/flan-t5-xl-sharded-bf16", torch_dtype="auto")
print(t5_model.dtype)  # torch.bfloat16
```


### Expected behavior

`AutoModelForSeq2SeqLM` should also load the model in `torch.bfloat16`. 
cc @younesbelkada <|||||>Hi @eladsegal 👋
![image](https://github.com/huggingface/transformers/assets/95997298/ee869958-221f-4605-a741-ef8058d32cb0)
- The model was saved in `bfloat16` with `"T5ForConditionalGeneration"` architecture so the model was loaded in `bfloat16`
-  But in `AutoModel.from_pretrained ` method torch_dtype is set to `auto` and you can read the doc ( image i have uploaded) that dtype picked by `auto` is generally `float32`
- Hope it helps , If i misunderstood your question ,Please 🙏 give a feedback

**You can check this --> [doc](https://huggingface.co/docs/transformers/v4.29.1/en/main_classes/configuration#transformers.PretrainedConfig.torch_dtype)**<|||||>Thank you for the report, @eladsegal - that's indeed a bug that I introduced while trying to fix another issue. 

Please try this fix: https://github.com/huggingface/transformers/pull/23379<|||||>Thank you @stas00 for the quick fix! Works just as expected. <|||||>Thank you for confirming that, @eladsegal!