one good next step would be to run per-op profiling (PyTorch Profiler — PyTorch Tutorials 1.8.0 documentation 7) on both the fp32 and quantized versions of your model, to see which kernels are contributing the most to inference time