Solved by khaidoan25 in post #5 
              
              
                Actually, when you load your quantized model, you need to quantize your initial model first. 
quantized_model = torch.quantization.quantize_dynamic(
    model, {nn.LSTM, nn.Linear}, dtype=torch.qint8
)         // Do s.t like this first before loading your quantized model