Loading a pytorch quantized model
Hi, I quantized a pre-trained model via **torch.quantization.quantize_dynamic** and saved it using **save_pretrained**.
based on https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/dynamic_quantization_bert_tutorial.ipynb#scrollTo=foe-dVxHIgOC . 
On reloading it using **from_pretrained** again the model blows up to its original size and the resultant predictions are garbage as well.
Is there a way to properly save the quantized weights and reload them?
Opening an issue because I couldn't find a resolution for the same.
@mfuntowicz @VictorSanh @patrickvonplaten 
Hello @amanpreet692 
Are you making sure you are reloading the saved (quantized) weights into a quantized model?
A dense checkpoint and its quantized counterpart have two different state dicts and I suspect you are re-initializing random matrices because when loading into from_pretrained it doesn't find the right keys in the state dict.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
