Here is the output from the autograd profiler before and after quantization:
Before
-----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
Name                     Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls  Input Shapes                         
-----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
conv2d                   0.67%            906.899us        0.67%            906.899us        906.899us        1                []                                   
convolution              0.67%            902.559us        0.67%            902.559us        902.559us        1                []                                   
_convolution             0.67%            900.649us        0.67%            900.649us        900.649us        1                []                                   
contiguous               0.00%            1.060us          0.00%            1.060us          1.060us          1                []                                   
contiguous               0.00%            0.140us          0.00%            0.140us          0.140us          1                []                                   
contiguous               0.00%            0.120us          0.00%            0.120us          0.120us          1                []                                   
mkldnn_convolution       0.66%            885.439us        0.66%            885.439us        885.439us        1                []                                   
conv2d                   0.53%            711.017us        0.53%            711.017us        711.017us        1                []                                   
convolution              0.53%            710.247us        0.53%            710.247us        710.247us        1                []                                   
_convolution             0.52%            704.057us        0.52%            704.057us        704.057us        1                []                                   
contiguous               0.00%            0.290us          0.00%            0.290us          0.290us          1                []                                   
contiguous               0.00%            0.090us          0.00%            0.090us          0.090us          1                []                                   
contiguous               0.00%            0.100us          0.00%            0.100us          0.100us          1                []                                   
mkldnn_convolution       0.52%            698.567us        0.52%            698.567us        698.567us        1                []                                   
conv2d                   0.29%            389.754us        0.29%            389.754us        389.754us        1                []                                   
convolution              0.29%            389.274us        0.29%            389.274us        389.274us        1                []                                   
_convolution             0.29%            388.564us        0.29%            388.564us        388.564us        1                []                                   
contiguous               0.00%            0.340us          0.00%            0.340us          0.340us          1                []                                   
contiguous               0.00%            0.090us          0.00%            0.090us          0.090us          1                []                                   
contiguous               0.00%            0.100us          0.00%            0.100us          0.100us          1                []                                   
mkldnn_convolution       0.29%            384.324us        0.29%            384.324us        384.324us        1                []                                   
relu_                    0.02%            29.550us         0.02%            29.550us         29.550us         1                []                                   
conv2d                   0.34%            454.195us        0.34%            454.195us        454.195us        1                []                                   
convolution              0.34%            453.735us        0.34%            453.735us        453.735us        1                []                                   
_convolution             0.34%            453.145us        0.34%            453.145us        453.145us        1                []                                   
contiguous               0.00%            0.240us          0.00%            0.240us          0.240us          1                []                                   
contiguous               0.00%            0.100us          0.00%            0.100us          0.100us          1                []                                   
contiguous               0.00%            0.090us          0.00%            0.090us          0.090us          1                []                                   
mkldnn_convolution       0.33%            448.975us        0.33%            448.975us        448.975us        1                []                                   
relu_                    0.02%            21.830us         0.02%            21.830us         21.830us         1                []                                   
conv2d                   0.22%            291.363us        0.22%            291.363us        291.363us        1                []                                   
convolution              0.22%            290.863us        0.22%            290.863us        290.863us        1                []                                   
_convolution             0.22%            290.223us        0.22%            290.223us        290.223us        1                []                                   
contiguous               0.00%            0.220us          0.00%            0.220us          0.220us          1                []                                   
contiguous               0.00%            0.100us          0.00%            0.100us          0.100us          1                []                                   
contiguous               0.00%            0.180us          0.00%            0.180us          0.180us          1                []                                   
mkldnn_convolution       0.21%            280.402us        0.21%            280.402us        280.402us        1                []                                   
adaptive_avg_pool2d      0.04%            60.060us         0.04%            60.060us         60.060us         1                []                                   
contiguous               0.00%            0.250us          0.00%            0.250us          0.250us          1                []                                   
view                     0.00%            5.270us          0.00%            5.270us          5.270us          1                []                                   
mean                     0.03%            44.300us         0.03%            44.300us         44.300us         1                []                                   
view                     0.00%            1.870us          0.00%            1.870us          1.870us          1                []                                   
view                     0.00%            1.690us          0.00%            1.690us          1.690us          1                []                                   
unsigned short           0.00%            5.701us          0.00%            5.701us          5.701us          1                []                                   
matmul                   0.03%            40.580us         0.03%            40.580us         40.580us         1                []                                   
mm                       0.03%            34.970us         0.03%            34.970us         34.970us         1                []                                   
relu_                    0.00%            3.950us          0.00%            3.950us          3.950us          1                []                                   
unsigned short           0.00%            2.340us          0.00%            2.340us          2.340us          1                []                                   
matmul                   0.00%            4.830us          0.00%            4.830us          4.830us          1                []                                   
mm                       0.00%            4.360us          0.00%            4.360us          4.360us          1                []                                   
sigmoid                  0.01%            13.000us         0.01%            13.000us         13.000us         1                []                                   
view                     0.00%            2.561us          0.00%            2.561us          2.561us          1                []                                   
expand_as                0.00%            4.660us          0.00%            4.660us          4.660us          1                []                                   
expand                   0.00%            3.220us          0.00%            3.220us          3.220us          1                []                                   
mul                      0.02%            21.070us         0.02%            21.070us         21.070us         1                []                                   
relu_                    0.01%            8.960us          0.01%            8.960us          8.960us          1                []                                   
conv2d                   0.21%            286.703us        0.21%            286.703us        286.703us        1                []                                   
convolution              0.21%            286.053us        0.21%            286.053us        286.053us        1                []                                   
_convolution             0.21%            285.113us        0.21%            285.113us        285.113us        1                []                                   
contiguous               0.00%            0.230us          0.00%            0.230us          0.230us          1                []                                   
contiguous               0.01%            17.500us         0.01%            17.500us         17.500us         1                []                                   
contiguous               0.00%            0.200us          0.00%            0.200us          0.200us          1                []                                   
mkldnn_convolution       0.20%            263.112us        0.20%            263.112us        263.112us        1                []                                   
conv2d                   0.33%            443.374us        0.33%            443.374us        443.374us        1                []                                   
convolution              0.33%            442.864us        0.33%            442.864us        442.864us        1                []                                   
_convolution             0.33%            442.304us        0.33%            442.304us        442.304us        1                []                                   
contiguous               0.00%            0.260us          0.00%            0.260us          0.260us          1                []                                   
contiguous               0.00%            0.100us          0.00%            0.100us          0.100us          1                []                                   
contiguous               0.00%            0.090us          0.00%            0.090us          0.090us          1                []                                   
mkldnn_convolution       0.33%            438.134us        0.33%            438.134us        438.134us        1                []                                   
relu_                    0.02%            27.920us         0.02%            27.920us         27.920us         1                []                                   
conv2d                   0.23%            310.863us        0.23%            310.863us        310.863us        1                []                                   
convolution              0.23%            310.383us        0.23%            310.383us        310.383us        1                []                                   
_convolution             0.23%            309.743us        0.23%            309.743us        309.743us        1                []                                   
contiguous               0.00%            0.230us          0.00%            0.230us          0.230us          1                []                                   
contiguous               0.00%            0.090us          0.00%            0.090us          0.090us          1                []                                   
contiguous               0.00%            0.170us          0.00%            0.170us          0.170us          1                []                                   
mkldnn_convolution       0.23%            305.503us        0.23%            305.503us        305.503us        1                []                                   
relu_                    0.01%            14.660us         0.01%            14.660us         14.660us         1                []                                   
conv2d                   0.19%            261.423us        0.19%            261.423us        261.423us        1                []                                   
convolution              0.19%            260.713us        0.19%            260.713us        260.713us        1                []                                   
_convolution             0.19%            255.493us        0.19%            255.493us        255.493us        1                []                                   
contiguous               0.00%            0.260us          0.00%            0.260us          0.260us          1                []                                   
contiguous               0.00%            0.120us          0.00%            0.120us          0.120us          1                []                                   
contiguous               0.00%            0.120us          0.00%            0.120us          0.120us          1                []                                   
mkldnn_convolution       0.19%            250.603us        0.19%            250.603us        250.603us        1                []                                   
conv2d                   0.20%            263.663us        0.20%            263.663us        263.663us        1                []                                   
convolution              0.20%            263.183us        0.20%            263.183us        263.183us        1                []                                   
_convolution             0.20%            262.683us        0.20%            262.683us        262.683us        1                []                                   
contiguous               0.00%            0.280us          0.00%            0.280us          0.280us          1                []                                   
contiguous               0.00%            0.100us          0.00%            0.100us          0.100us          1                []                                   
contiguous               0.00%            0.090us          0.00%            0.090us          0.090us          1                []                                   
mkldnn_convolution       0.19%            258.533us        0.19%            258.533us        258.533us        1                []                                   
relu_                    0.01%            13.400us         0.01%            13.400us         13.400us         1                []                                   
conv2d                   0.15%            196.892us        0.15%            196.892us        196.892us        1                []                                   
convolution              0.15%            196.412us        0.15%            196.412us        196.412us        1                []                                   
_convolution             0.15%            195.812us        0.15%            195.812us        195.812us        1                []                                   
contiguous               0.00%            0.240us          0.00%            0.240us          0.240us          1                []                                   
contiguous               0.00%            0.110us          0.00%            0.110us          0.110us          1                []                                   
contiguous               0.00%            0.100us          0.00%            0.100us          0.100us          1                []                                   
-----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
Self CPU time total: 134.621ms