hit the character limit…
After
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
Name                         Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls  Input Shapes                         
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
item                         0.01%            7.420us          0.01%            7.420us          7.420us          1                []                                   
_local_scalar_dense          0.01%            4.550us          0.01%            4.550us          4.550us          1                []                                   
aten::Int                    0.00%            1.720us          0.00%            1.720us          1.720us          1                []                                   
item                         0.00%            0.470us          0.00%            0.470us          0.470us          1                []                                   
_local_scalar_dense          0.00%            0.240us          0.00%            0.240us          0.240us          1                []                                   
quantize_per_tensor          0.09%            69.811us         0.09%            69.811us         69.811us         1                []                                   
quantized::conv2d            1.53%            1.183ms          1.53%            1.183ms          1.183ms          1                []                                   
contiguous                   0.22%            167.211us        0.22%            167.211us        167.211us        1                []                                   
empty_like                   0.01%            9.810us          0.01%            9.810us          9.810us          1                []                                   
qscheme                      0.00%            0.920us          0.00%            0.920us          0.920us          1                []                                   
q_zero_point                 0.00%            0.710us          0.00%            0.710us          0.710us          1                []                                   
q_scale                      0.00%            0.740us          0.00%            0.740us          0.740us          1                []                                   
_empty_affine_quantized      0.00%            2.930us          0.00%            2.930us          2.930us          1                []                                   
q_scale                      0.00%            0.170us          0.00%            0.170us          0.170us          1                []                                   
contiguous                   0.00%            0.160us          0.00%            0.160us          0.160us          1                []                                   
_empty_affine_quantized      0.00%            1.440us          0.00%            1.440us          1.440us          1                []                                   
quantize_per_tensor          0.01%            5.550us          0.01%            5.550us          5.550us          1                []                                   
_empty_affine_quantized      0.00%            1.320us          0.00%            1.320us          1.320us          1                []                                   
q_zero_point                 0.00%            0.180us          0.00%            0.180us          0.180us          1                []                                   
q_scale                      0.00%            0.150us          0.00%            0.150us          0.150us          1                []                                   
q_zero_point                 0.00%            0.150us          0.00%            0.150us          0.150us          1                []                                   
q_scale                      0.00%            0.130us          0.00%            0.130us          0.130us          1                []                                   
quantized::conv2d            0.34%            266.002us        0.34%            266.002us        266.002us        1                []                                   
contiguous                   0.00%            0.140us          0.00%            0.140us          0.140us          1                []                                   
q_scale                      0.00%            0.160us          0.00%            0.160us          0.160us          1                []                                   
contiguous                   0.00%            0.100us          0.00%            0.100us          0.100us          1                []                                   
_empty_affine_quantized      0.00%            1.260us          0.00%            1.260us          1.260us          1                []                                   
quantize_per_tensor          0.01%            4.290us          0.01%            4.290us          4.290us          1                []                                   
_empty_affine_quantized      0.00%            1.180us          0.00%            1.180us          1.180us          1                []                                   
q_zero_point                 0.00%            0.180us          0.00%            0.180us          0.180us          1                []                                   
q_scale                      0.00%            0.150us          0.00%            0.150us          0.150us          1                []                                   
q_zero_point                 0.00%            0.140us          0.00%            0.140us          0.140us          1                []                                   
q_scale                      0.00%            0.140us          0.00%            0.140us          0.140us          1                []                                   
quantized::conv2d_relu       1.11%            856.897us        1.11%            856.897us        856.897us        1                []                                   
contiguous                   0.00%            0.170us          0.00%            0.170us          0.170us          1                []                                   
q_scale                      0.00%            0.160us          0.00%            0.160us          0.160us          1                []                                   
contiguous                   0.00%            0.150us          0.00%            0.150us          0.150us          1                []                                   
_empty_affine_quantized      0.00%            1.260us          0.00%            1.260us          1.260us          1                []                                   
quantize_per_tensor          0.01%            4.370us          0.01%            4.370us          4.370us          1                []                                   
_empty_affine_quantized      0.00%            1.270us          0.00%            1.270us          1.270us          1                []                                   
q_zero_point                 0.00%            0.170us          0.00%            0.170us          0.170us          1                []                                   
q_scale                      0.00%            0.150us          0.00%            0.150us          0.150us          1                []                                   
q_zero_point                 0.00%            0.150us          0.00%            0.150us          0.150us          1                []                                   
q_scale                      0.00%            0.130us          0.00%            0.130us          0.130us          1                []                                   
quantized::conv2d_relu       0.49%            378.753us        0.49%            378.753us        378.753us        1                []                                   
contiguous                   0.00%            0.200us          0.00%            0.200us          0.200us          1                []                                   
q_scale                      0.00%            0.170us          0.00%            0.170us          0.170us          1                []                                   
contiguous                   0.00%            0.140us          0.00%            0.140us          0.140us          1                []                                   
_empty_affine_quantized      0.00%            1.290us          0.00%            1.290us          1.290us          1                []                                   
quantize_per_tensor          0.01%            4.700us          0.01%            4.700us          4.700us          1                []                                   
_empty_affine_quantized      0.00%            1.260us          0.00%            1.260us          1.260us          1                []                                   
q_zero_point                 0.00%            0.170us          0.00%            0.170us          0.170us          1                []                                   
q_scale                      0.00%            0.150us          0.00%            0.150us          0.150us          1                []                                   
q_zero_point                 0.00%            0.140us          0.00%            0.140us          0.140us          1                []                                   
q_scale                      0.00%            0.130us          0.00%            0.130us          0.130us          1                []                                   
quantized::conv2d            0.18%            140.401us        0.18%            140.401us        140.401us        1                []                                   
contiguous                   0.00%            0.180us          0.00%            0.180us          0.180us          1                []                                   
q_scale                      0.00%            0.160us          0.00%            0.160us          0.160us          1                []                                   
contiguous                   0.00%            0.100us          0.00%            0.100us          0.100us          1                []                                   
_empty_affine_quantized      0.00%            1.060us          0.00%            1.060us          1.060us          1                []                                   
quantize_per_tensor          0.01%            3.920us          0.01%            3.920us          3.920us          1                []                                   
_empty_affine_quantized      0.00%            1.260us          0.00%            1.260us          1.260us          1                []                                   
q_zero_point                 0.00%            0.170us          0.00%            0.170us          0.170us          1                []                                   
q_scale                      0.00%            0.160us          0.00%            0.160us          0.160us          1                []                                   
q_zero_point                 0.00%            0.150us          0.00%            0.150us          0.150us          1                []                                   
q_scale                      0.00%            0.140us          0.00%            0.140us          0.140us          1                []                                   
size                         0.00%            0.970us          0.00%            0.970us          0.970us          1                []                                   
size                         0.00%            0.190us          0.00%            0.190us          0.190us          1                []                                   
adaptive_avg_pool2d          0.02%            16.650us         0.02%            16.650us         16.650us         1                []                                   
_adaptive_avg_pool2d         0.02%            14.410us         0.02%            14.410us         14.410us         1                []                                   
view                         0.01%            4.641us          0.01%            4.641us          4.641us          1                []                                   
quantized::linear            0.02%            14.480us         0.02%            14.480us         14.480us         1                []                                   
contiguous                   0.00%            0.160us          0.00%            0.160us          0.160us          1                []                                   
q_scale                      0.00%            0.340us          0.00%            0.340us          0.340us          1                []                                   
_empty_affine_quantized      0.00%            1.010us          0.00%            1.010us          1.010us          1                []                                   
quantize_per_tensor          0.01%            4.510us          0.01%            4.510us          4.510us          1                []                                   
_empty_affine_quantized      0.00%            0.930us          0.00%            0.930us          0.930us          1                []                                   
q_scale                      0.00%            0.220us          0.00%            0.220us          0.220us          1                []                                   
q_zero_point                 0.00%            0.200us          0.00%            0.200us          0.200us          1                []                                   
relu_                        0.00%            3.630us          0.00%            3.630us          3.630us          1                []                                   
quantized::linear            0.01%            10.470us         0.01%            10.470us         10.470us         1                []                                   
contiguous                   0.00%            0.150us          0.00%            0.150us          0.150us          1                []                                   
q_scale                      0.00%            0.170us          0.00%            0.170us          0.170us          1                []                                   
_empty_affine_quantized      0.00%            0.850us          0.00%            0.850us          0.850us          1                []                                   
quantize_per_tensor          0.01%            4.310us          0.01%            4.310us          4.310us          1                []                                   
_empty_affine_quantized      0.00%            0.810us          0.00%            0.810us          0.810us          1                []                                   
q_scale                      0.00%            0.180us          0.00%            0.180us          0.180us          1                []                                   
q_zero_point                 0.00%            0.150us          0.00%            0.150us          0.150us          1                []                                   
sigmoid                      0.01%            7.770us          0.01%            7.770us          7.770us          1                []                                   
view                         0.00%            1.290us          0.00%            1.290us          1.290us          1                []                                   
expand_as                    0.01%            5.330us          0.01%            5.330us          5.330us          1                []                                   
expand                       0.01%            4.190us          0.01%            4.190us          4.190us          1                []                                   
quantized::mul               0.20%            152.521us        0.20%            152.521us        152.521us        1                []                                   
qscheme                      0.00%            0.290us          0.00%            0.290us          0.290us          1                []                                   
qscheme                      0.00%            0.180us          0.00%            0.180us          0.180us          1                []                                   
qscheme                      0.00%            0.140us          0.00%            0.140us          0.140us          1                []                                   
_empty_affine_quantized      0.00%            1.380us          0.00%            1.380us          1.380us          1                []                                   
q_zero_point                 0.00%            0.170us          0.00%            0.170us          0.170us          1                []                                   
q_scale                      0.00%            0.200us          0.00%            0.200us          0.200us          1                []                                   
q_zero_point                 0.00%            0.140us          0.00%            0.140us          0.140us          1                []                                   
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
Self CPU time total: 77.276ms

Most of the time is spent in Conv2d/ReLU operations, but they are quantized. So it seems that quantization is indeed working as Desktop CPU time decreases from 134ms to 77ms.
However, when I run the quantized model on my mobile device (Huawei Mate 10 lite), there are no performance gains. Any ideas?
Also, I found that the noise is likely caused by this qnnpack bug https://github.com/pytorch/pytorch/issues/36253 1. When I train my model only for a few epochs, I can quantize the model without any errors. However, when I fully train the model, these errors: “output scale: convolution scale 4.636909 is greater or equal to 1.0” are thrown during quantization and the model is extremely noisy after quantization. Is there a fix for this yet?