Solved by dskhudia in post #5 
              
              
                @wizardk: Thanks for reporting it. It’s an issue in our backend library, FBGEMM, for certain specific depthwise shapes (i_c != o_c ). In your case, depthwise convolution goes through a slower path. https://github.com/pytorch/FBGEMM/issues/347 is tracking the progress on improving performance for suc…