Solved by Giang_Dang in post #6 
              
              
                Hi mohit7, 
Make sure you create the net using previous definition, and let the net go through process that was applied during quantization before (prepare_model, fuse_model, and convert), without rerun the calibration process. 
After that you can load the quantized state_dict in. Hope it helps.