figured it out here: https://github.com/pytorch/pytorch/issues/30622 172

Assuming that you are indeed compiling PyTorch with NNPACK enabled … you are effectively at the mercy of the the checks in aten/src/ATen/native/Convolution.cpp:_convolution() for convolutions (and equivalent checks for other operators) which do their best to pick the most efficient code path based on a multitude of conditions, the core of which usually boil down to testing whether an efficient implementation indeed exists for that particular configuration or not.  Your best bet then, would be to either put a breakpoint in that function or printf-debug it to see why a more efficient code path is not taken, and then modify your network to meet those requirements.  It might also be the case that you are not compiling with NNPACK enabled.