Hi, I have quantized a MobileNetV3-like Network with ‘qnnpack’ for use in an Android app. However, the quantized model is even slower than the original one.
All layers seem to be quantized correctly and the model file size decreased to 1/4 of the original size.
The model has ~2M Parameters and input resolution is 224x224.
Here are some inference time numbers:
Model (without quantization) on Ryzen 3700x: ~50ms
Model (without quantization) on RTX 2070: ~6ms
Model (without quantization) on Huawei Mate 10 lite: ~1s
Model (with quantization) on Huawei Mate 10 lite: ~1.5s
I did not expect that inference would take ~1s on such a model, even without quantization. Is this expected?
Why would a quantized model be slower? Are there any operations/layers/architecture conventions that should be absolutely avoided?
Also, the output of the quantized model is extremely noisy. What could be causing this?
Here is an output example before and after model quantization:
noise1500×732 260 KB