FYI, in order to investigate the bottleneck of model execution, I profiled my quantized model using torch.autograd.profiler.profile().
I thought there is some problem about quantized::conv2d which QNNPACK supports.
I will share the further progress, thank you so much @Vasiliy_Kuznetsov ! 
 ---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                         Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
quantized::conv2d            92.64%           2.221s           92.67%           2.222s           170.929ms        13
quantized::linear            5.69%            136.383ms        5.69%            136.452ms        45.484ms         3
_adaptive_avg_pool2d         0.64%            15.370ms         0.64%            15.370ms         15.370ms         1
relu_                        0.49%            11.650ms         0.49%            11.650ms         776.697us        15
quantized_max_pool2d         0.40%            9.491ms          0.40%            9.491ms          1.898ms          5
quantize_per_tensor          0.09%            2.261ms          0.09%            2.261ms          2.261ms          1
contiguous                   0.02%            410.239us        0.02%            450.143us        28.134us         16
_empty_affine_quantized      0.01%            304.640us        0.01%            304.640us        17.920us         17
max                          0.01%            160.672us        0.01%            160.672us        160.672us        1
q_scale                      0.00%            113.983us        0.00%            113.983us        2.478us          46
clone                        0.00%            102.719us        0.00%            102.719us        102.719us        1
dequantize                   0.00%            50.016us         0.00%            50.016us         50.016us         1
q_zero_point                 0.00%            45.728us         0.00%            45.728us         1.524us          30
view                         0.00%            44.704us         0.00%            44.704us         44.704us         1
max_pool2d                   0.00%            36.128us         0.40%            9.527ms          1.905ms          5
select                       0.00%            31.680us         0.00%            31.680us         31.680us         1
reshape                      0.00%            30.208us         0.01%            195.071us        97.535us         2
_unsafe_view                 0.00%            17.440us         0.00%            17.440us         17.440us         1
empty_like                   0.00%            13.888us         0.00%            39.904us         39.904us         1
_local_scalar_dense          0.00%            13.504us         0.00%            13.504us         4.501us          3
is_floating_point            0.00%            13.440us         0.00%            13.440us         13.440us         1
item                         0.00%            12.448us         0.00%            25.952us         8.651us          3
flatten                      0.00%            7.456us          0.01%            138.463us        138.463us        1
adaptive_avg_pool2d          0.00%            5.312us          0.64%            15.376ms         15.376ms         1
dropout                      0.00%            5.216us          0.00%            5.216us          2.608us          2
qscheme                      0.00%            4.736us          0.00%            4.736us          4.736us          1
is_complex                   0.00%            3.360us          0.00%            3.360us          3.360us          1
sizes                        0.00%            2.656us          0.00%            2.656us          2.656us          1
size                         0.00%            2.240us          0.00%            2.240us          2.240us          1
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------