Loading fp16 model as fp32 when using .from_retrained()
### System Info


When loading GPTJ from GPTJForCausalLM.from_pretrained() loading 16bit model which should be approximately 12GB, it instead has a model size ~23GB which is the full 32 bit weights.

the code:
```
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

tokenizer = AutoTokenizer.from_pretrained("ThisIsMyUsername69/gpt-j-6B-16bit")
config = AutoConfig.from_pretrained("ThisIsMyUsername69/gpt-j-6B-16bit", torch_dtype=torch.float16)
model = GPTJForCausalLM.from_pretrained("ThisIsMyUsername69/gpt-j-6B-16bit", config=config)
```
I've tried multiple ways of trying to load in 16 bit, from_config, with or without autoconfig, regardless of everything it seems to always use 23GB of VRAM except with EleutherAI/gpt-j-6B using revision float16.

The model has memory footprint of 23194MiB.  ="ThisIsMyUsername69/gpt-j-6B-16bit" and "nlpcloud/instruct-gpt-j-fp16" both give the larger model size.

I have tried giving the parameter ' revision="float16" ' and it gives the same response.  

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. run the code above
2. observe model size

### Expected behavior

The model size should be approximately 11GB however it is giving the full model weight (32 bit float) size.
cc @younesbelkada <|||||>Hi @EdanZizo 
to load the model with the desired dtype directly from the config I believe you should use `torch_dtype="auto"` in `GPTJForCausalLM.from_pretrained("ThisIsMyUsername69/gpt-j-6B-16bit", config=config)`. But note that the canonical way to load any model in half precision is: 
```python
GPTJForCausalLM.from_pretrained("ThisIsMyUsername69/gpt-j-6B-16bit", torch_dtype=torch.float16)
```<|||||>It is working now, thanks for your help.