Quantized model not preserved when imported using from_pretrained()
## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
When I import the saved quantized model using `from_pretrained()`, the model's size is inflated to the pre-quantized version.  The model also takes a significant performance hit, both accuracy and time, from the original quantized model.

The tasks I am working on is:
This is the official Pytorch notebook from: https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html

https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/dynamic_quantization_bert_tutorial.ipynb

## To Reproduce

Steps to reproduce the behavior:

1.  Run through the notebook fully.
2.  Load the quantized model with from_pretrained()
3.  Run `3.1 Check the model size` and `3.2 Evaluate the inference accuracy and time`

You will see that the size is of the pre-quantized model (>400 MB), and the accuracy AND time take a huge hit from the original quantized model.

## Expected behavior

1.  The quantized model can be loaded in at its original size of <200 MB
2.  The quantized model preserves its accuracy when loaded in
3.  The quantized model preserves its time to run when loaded in

## Environment

Colab

@LysandreJik any ideas on this?  I am itching to use a quantized BERT model in production, but it does not work when loaded in :(<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hi @ElektrikSpark , I am also facing similar issue, were you able to resolve this? 
I am unable to get good results while loading the quantized bert model. 