Thanks!
I tried checking the quantization error for each of the layers and got the following:
conv_stem.0.weight tensor(44.4819)
blocks.0.0.conv_dw.0.weight tensor(45.0884)
blocks.0.0.conv_pw.weight tensor(42.8196)
blocks.1.0.conv_pw.0.weight tensor(43.1310)
blocks.1.0.conv_dw.0.weight tensor(46.9183)
blocks.1.0.conv_pwl.weight tensor(43.1703)
blocks.1.1.conv_pw.0.weight tensor(44.4646)
blocks.1.1.conv_dw.0.weight tensor(45.7783)
blocks.1.1.conv_pwl.weight tensor(39.9211)
blocks.2.0.conv_pw.0.weight tensor(44.0625)
blocks.2.0.conv_dw.0.weight tensor(45.3749)
blocks.2.0.conv_pwl.weight tensor(41.9430)
blocks.2.1.conv_pw.0.weight tensor(43.8883)
blocks.2.1.conv_dw.0.weight tensor(42.4965)
blocks.2.1.conv_pwl.weight tensor(40.5602)
blocks.3.0.conv_pw.0.weight tensor(43.9803)
blocks.3.0.conv_dw.0.weight tensor(47.7440)
blocks.3.0.conv_pwl.weight tensor(41.9959)
blocks.3.1.conv_pw.0.weight tensor(43.2630)
blocks.3.1.conv_dw.0.weight tensor(45.7537)
blocks.3.1.conv_pwl.weight tensor(41.7492)
blocks.3.2.conv_pw.0.weight tensor(43.5795)
blocks.3.2.conv_dw.0.weight tensor(45.5840)
blocks.3.2.conv_pwl.weight tensor(41.2215)
blocks.4.0.conv_pw.0.weight tensor(42.7768)
blocks.4.0.conv_dw.0.weight tensor(41.5424)
blocks.4.0.conv_pwl.weight tensor(41.2056)
blocks.4.1.conv_pw.0.weight tensor(43.2486)
blocks.4.1.conv_dw.0.weight tensor(43.3677)
blocks.4.1.conv_pwl.weight tensor(41.5483)
blocks.4.2.conv_pw.0.weight tensor(43.2695)
blocks.4.2.conv_dw.0.weight tensor(43.2045)
blocks.4.2.conv_pwl.weight tensor(41.8538)
blocks.5.0.conv_pw.0.weight tensor(42.5763)
blocks.5.0.conv_dw.0.weight tensor(46.0717)
blocks.5.0.conv_pwl.weight tensor(41.6060)
blocks.5.1.conv_pw.0.weight tensor(42.4102)
blocks.5.1.conv_dw.0.weight tensor(44.6428)
blocks.5.1.conv_pwl.weight tensor(40.9154)
blocks.5.2.conv_pw.0.weight tensor(42.4992)
blocks.5.2.conv_dw.0.weight tensor(44.1465)
blocks.5.2.conv_pwl.weight tensor(40.3739)
blocks.5.3.conv_pw.0.weight tensor(42.2826)
blocks.5.3.conv_dw.0.weight tensor(44.1184)
blocks.5.3.conv_pwl.weight tensor(40.7068)
blocks.6.0.conv_pw.0.weight tensor(42.2656)
blocks.6.0.conv_dw.0.weight tensor(47.4642)
blocks.6.0.conv_pwl.weight tensor(41.3921)
conv_head.0.weight tensor(42.7725)
classifier._packed_params._packed_params tensor(39.3391)


I’m not sure if these are large values or standard for this type of quantization, but it doesn’t seem that a specific layer is significantly worse than others.
Maybe it is something regarding the AdaptiveAvgPool2d?
I saw there were changes to it in the release notes.