Sadly no 
I’m facing same issue on UNet architecture
What I’ve found out is that when passing smaller input like (1, 3, 32, 32) quantized model performs similarly to fp32. When passing even smaller input like (1,3,16,16) quantized model performs slightly better (6% speedup on fbgemm). Nonetheless input of size 16x16 is quite extreme scenario.
@supriyar @dskhudia maybe this is helpful to trace what is wrong?