Hi,
I am trying to use QAT to speed up a segmentation model on CPU.
The preparation, training and conversion to a quantized model all seem to work fine: negligible drop in performance and reduction in model size by ~4.
However, I am getting some strange latency measurements with the quantized model where larger image take more time for inference than the original model.
Here are a few numbers for a MobileNetV3 large with dilation and reduced tail (see Everything you need to know about TorchVision’s MobileNetV3 implementation | PyTorch) with the LR-ASPP head on top for the segmentation:

Fused model CPU latency:

256x256: 76 ms
512x512: 206 ms
1024x1024: 706 ms


Quantized model CPU latency:

256x256: 53 ms
512x512: 211 ms
1024x1024: 849 ms



These numbers were obtained with torch.set_num_threads(4) on a Ryzen 7 3700X.
For some reason, at higher resolutions, the model is slower with quantization. I am also using torchvision’s implementation of quantizable MobileNetV3 (vision/mobilenetv3.py at master · pytorch/vision · GitHub).
Any idea where this could come from?