@ptrblck just to give you update  regarding dynamic quantization on cpu. There is issue with quantized bert model which I and many others are facing. Here is the github issue link : https://github.com/huggingface/transformers/issues/2542 19