I can no longer find the requisite doc link, but Pytorch dynamic quantization is currently (v1.5) only provided for Linear and LSTM layers. A model that doesn’t have a high proportion of those will not benefit from dynamic quantization.
(To confirm, print() a quantized model to see which layers have been replaced.)