Saving a 8-bit T5 does not work.
### System Info

Latest.

### Who can help?

@patrickvonplaten 

I am able to load a T5 model in 8-bit-format using this command:
`model = T5ForConditionalGeneration.from_pretrained(".",load_in_8bit=True, device_map="auto")`

The model works fine. Saving the model using ".save_pretrained()" also seems to work, and the file on the disk is a lot smaller. I am able to load the save model again without errors, but then the model no longer works.

Here are two sample models:
north/fine_North_large
north/fine_North_large_8bit


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

See description above

### Expected behavior

The model should not change when being saved.
Those are all arguments for inference specifically. `save_pretrained` does not work if you use `device_map="auto"` if you have weights offloaded on the CPU (not sure if it's the case or not here) and it's not been tested with `load_in_8bit` either.

We will probably fix this later down the road, but for now you should not save on disk the model this way (you can find the model in your cache if you need its weights anyway)<|||||>Thanks for the feedback, @sgugger.

So, just to be sure that I understand this: There is not really any way of easily distributing only the 8-bit model. The correct way of doing it will be to distribute the 32-bit version, and then ask people to use **load_in_8bit=True, device_map="auto"** ?

What about loading the 8-bit-version in the HuggingFace widgets. Is that even possible?<|||||>There is no way to save and reload an 8-nit model anyway (correct me if I'm wrong @younesbelkada ) so you need to use `load_in_8bit=True, device_map="auto"` indeed.<|||||>Yes this is correct, you can also save the model in fp16 to save memory and load it back in int8, but currently saving and loading the model in 8-bit is not supported