torch_dtype='auto' is not working when using AutoModel.from_pretrained(...)
### System Info

- `transformers` version: 4.29.2
- Platform: macOS-12.2.1-x86_64-i386-64bit
- Python version: 3.11.3
- Huggingface_hub version: 0.15.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@younesbelkada @ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The `torch_dtype='auto'` argument is not forwarded correctly when using `AutoModelForCausalLM(model_name, torch_dtype='auto')`. For example, the opt model 'facebook/opt-125m' has `torch_dtype: float16` in its config file but the following happens:

```python
from transformers import AutoModelForCausalLM, OPTForCausalLM

model = OPTForCausalLM.from_pretrained('facebook/opt-125m', torch_dtype='auto')
model.dtype
# CORRECT dtype
>>> torch.float16

model = AutoModelForCausalLM.from_pretrained('facebook/opt-125m', torch_dtype='auto')
model.dtype
# INCORRECT dtype 
>>> torch.float32
```

### Expected behavior

Both outputs should be `torch.float16`  as in the config file specification.

 From what I've looked into, this comes from 
```python
if kwargs_copy.get("torch_dtype", None) == "auto":
    _ = kwargs_copy.pop("torch_dtype")
```

in `transformers.models.auto.auto_factory.py`, line 441. The additional kwarg specifying dtype is poped and the dtype is only inferred by the dtype argument of the config file, which is then not given explicitly (only implicitly in the config) to `PretrainedModel.from_pretrained(model_name, config=config,...)`, which does not use it if the explicit `torch_dtype` argument is not provided.

I would be happy to help solve the issue if needed. 

Also, I find it strange that
```python
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained('facebook/opt-125m')
config.torch_dtype
>>> torch.float16

model = AutoModelForCausalLM.from_pretrained('facebook/opt-125m', config=config)
model.dtype
>>> torch.float32
```

i.e. `AutoModelForCausalLM.from_pretrained(...)` does not respect the dtype of the config as I was saying before (this is the reason of the previous bug). But maybe this is to avoid errors when model configs specify `torch_dtype: bfloat16` and users try to instantiate on the cpu?

Anyway, when specifying 'auto', i.e. `AutoModelForCausalLM.from_pretrained(...torch_dtype='auto')`, I think it is absolutely necessary for the model to be instantiated with the dtype specified on the config file if any, even if it may break code for `torch.bfloat16` models, because users using this feature are aware of what they are doing.
Pinging @younesbelkada as I think he looked into this previously! 