As given in the tutorial here: https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html 6 The size reduction from 438M -> 181.5M matches. For the time, we expect approximately a 2x performance improvement on  Xeon E5-2620 v4. Can’t say what’s the issue without access to your env setup.