Fix BLOOM dtype
Issues:
a) The bloom attribute map does not follow the convention of 
```json
common_arg: model_specific_arg
```
& the argument n_embed is there for no apparent reason.

b) Using attributes with defaults in the attribute_map leads to them being overwritten in [setattr](https://github.com/huggingface/transformers/blob/7498db06a1724773ad7185887743afcb8406ee6b/src/transformers/configuration_utils.py#L249) when the respective kwarg is popped [here](https://github.com/huggingface/transformers/blob/7498db06a1724773ad7185887743afcb8406ee6b/src/transformers/configuration_utils.py#L265). For now I just removed the `dtype` argument for `torch_dtype`. I think the best solution for this would be to check the attribute map before popping. I can also implement that instead if someone gives me a heads up.

c) Weights end up in fp32 & not the torch_dtype when loading them in via load_state_dict. I havn't looked into this in more detail, so I just fixed it via casting, which is sufficient for the conversion script I think.




Notes: 
- If we merge this, we need to update all BLOOM configs in the Hub by swapping `dtype` for `torch_dtype` & swapping `n_embed` for `hidden_size`
_The documentation is not available anymore as the PR was closed or merged._<|||||>You should just set the `dtype` in the config file<|||||>Hmm maybe I'm doing sth wrong. I use

`python /gpfsssd/worksf/projects/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/transformers/models/bloom/convert_bloom_original_checkpoint_to_pytorch.py --bloom_checkpoint_path /gpfsscratch/rech/six/commun/checkpoints/tr11e-350M-ml/checkpoints/main/global_step659500 --pytorch_dump_folder_path bloomfp16 --bloom_config_file /gpfsscratch/rech/six/commun/commun/experiments/muennighoff/bloomckpt/350M/bloom-350m/config.json --pretraining_tp 1`

where the config is the below cloned from [here](https://huggingface.co/bigscience/bloom-350m)

```json
{
  "apply_residual_connection_post_layernorm": false,
  "attention_dropout": 0.0,
  "attention_softmax_in_fp32": true,
  "bias_dropout_fusion": true,
  "bos_token_id": 1,
  "dtype": "float16",
  "eos_token_id": 2,
  "pad_token_id": 3,
  "unk_token_id": 0,
  "hidden_dropout": 0.0,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "masked_softmax_fusion": true,
  "model_type": "bloom",
  "n_embed": 1024,
  "n_inner": null,
  "n_layer": 24,
  "num_attention_heads": 16,
  "offset_alibi": 100,
  "pretraining_tp": 1,
  "seq_length": 2048,
  "skip_bias_add": true,
  "skip_bias_add_qkv": false,
  "transformers_version": "4.20.0.dev0",
  "use_cache": true,
  "vocab_size": 250880
}
```<|||||>See updated description cc @thomasw21 , @younesbelkada <|||||>Thanks for the fix! Indeed there seems to be a mistake there, and since the native weights are already in `fp16` / `bfloat16` we did not observed it since the `torch_dtype` can be automatically retrieved here when doing `torch_dtype="auto"`: https://github.com/huggingface/transformers/blob/7498db06a1724773ad7185887743afcb8406ee6b/src/transformers/modeling_utils.py#L196 
I will ask @sgugger to review this since it may break backward compatibility but I think if we change all config files it should be fine <|||||>So in case we want to load models in half precision we always have to add `torch_dtype=auto` when loading a model using `from_pretrained` ?<|||||>Sure, I will remove `torch_dtype` then entirely from the Bloom config.
Like @younesbelkada said in order to get BLOOM in the correct dtype one will always have to specify `torch_dtype=auto`. The below returns a model in fp32 not the desired bf16:

```python
from transformers import AutoModel, BloomModel
model = BloomModel.from_pretrained("bigscience/bloom-350m")
```<|||||>Yes, but that was already the case before this PR.<|||||>Thanks for iterating! I just looked and all the configs online define `n_embed` instead of `hidden_size`. So we will need some backward compatible line in the init to pop `n_embed` from the kwargs and set it as hidden_size to not break everything.

We can't just replace the existing configs as it would break BLOOM with the current version of Transformers. The `dtype` field is different since it wasn't used anywhere.

Also, you will need a rebase on main to have the tests (example torch test specifically) finish faster :-)<|||||>Tests are passing & it is backwards compatible with `n_embed`. Let's merge?
I can remove `dtype` from the bloom configs under bigscience; Even if it's still set somewhere, it shouldn't be a problem as it hasn't been used before and would still just be an unused kwarg<|||||>Yes, we can merge, thanks! We can remove the `dtype` attributes in the configurations but the `n_embed` ones will need to stay for compatibility with older versions of Transformers.<|||||>I'm late to chime in. We absolutely need `torch_dtype` in all configs, since some models are fp16 and others are bf16. The user needs to know even if the core doesn't yet use `torch_dtype` directly. It can be used as:

`AutoModel.from_pretrained(..., torch_dtype=config.torch_dtype)`

model configs should ideally be created with `save_from_pretrained`, which will do the right thing, but given the size on some of the models, like 176B, we might have to find a workaround for that one.

And some models are missing crucial configs - hidden size.

Thank you!<|||||>> I'm late to chime in. We absolutely need `torch_dtype` in all configs, since some models are fp16 and others are bf16. The user needs to know even if the core doesn't yet use `torch_dtype` directly. It can be used as:
> 
> `AutoModel.from_pretrained(..., torch_dtype=config.torch_dtype)`
> 
> model configs should ideally be created with `save_from_pretrained`, which will do the right thing, but given the size on some of the models, like 176B, we might have to find a workaround for that one.
> 
> And some models are missing crucial configs - hidden size.
> 
> Thank you!

Good point. I think we can add the `torch_dtype` kwarg to the hub configs for information purposes, but it does not change the loading of the models.
I.e. 
```python
from transformers import AutoModel, AutoConfig

model = AutoModel.from_pretrained("Muennighoff/bloom-tiny-random")
model.dtype
```
yields `torch.float32` even though `torch_dtype` is [set](https://huggingface.co/Muennighoff/bloom-tiny-random/blob/main/config.json) to `float16`. I'm not sure if this is a feature or a bug?

So models always have to be loaded with

```python
from transformers import AutoModel, AutoConfig

model = AutoModel.from_pretrained("Muennighoff/bloom-tiny-random", torch_dtype="auto")
model.dtype # torch.bfloat16
```

<|||||>@Muennighoff the behaviour that you are exposing is expected, if you want to load a model with its native dtype you always have to pass `torch_dtype=auto`. I imagine the reason behind this choice is to let some operations (eg LayerNorm) that cannot run in half precision on CPU, be able to run them by default on CPU.<|||||>not **always**, sometimes one needs to instantiate a model w/o having access to weights, so one has to do:
```
AutoModel.from_config(config, torch_dtype=config.torch_dtype)
```

"auto" won't work here.

we had to use it for DeepSpeed-Inference just recently.<|||||>The main reason for `torch_dtype` not being used automatically on load is that most model configs don't have it. So we decided to start populating this meta data and eventually switch to using it. I think we added it a year ago or so.

Otherwise there would be an inconsistent behavior where some models would load in a correct dtype and other won't. So for now it's an explicit way.