I think kernel size = 4 and dewpthwise conv is the culprit here. Quanatized depthwise is optimized mainly for common kernel sizes 3 and 5. Just to reiterate @supriyar’s question: Is there any reason to use kernel  size 4?