Llama weights are in `bfloat16` but loaded as `float32`
### System Info

- `transformers` version: 4.32.0.dev0
- Platform: Linux-5.8.0-1035-gcp-x86_64-with-glibc2.31
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0.dev20230719+cpu (False)
- Tensorflow version (GPU?): 2.14.0-dev20230719 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (cpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
model = LlamaForCausalLM.from_pretrained(path)
print(model.model.norm.weight.detach().dtype)
```

This prints `torch.float32`.

### Expected behavior

This result is `torch.float32`, which indicates that the model is loaded in `float32` format. However, the result should be `torch.bfloat16` because it is the default for Llama models. The model weights are in `bfloat16` format.
Hey! 👋🏻 
Without the path to the checkpoint that you are using, it's gonna be a bit hard for me to debug. The `dtype` can be affectect by the `model.config.torch_dtype` which is stored online. <|||||>It is expected behavior: https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained.torch_dtype. You can try to pass `torch_dtype="auto"` to `from_pretrained` and if the config.json is well written, it should load in the dtype specified in the config.<|||||>In general you should specify the dtype in which you want the model with `torch_dtype` (using auto is not recommended as configs are not always well written). In PyTorch models are always loaded in float32 by default (even if the state dict has another dtype), this is not something specific to Transformers.