I think you’re right. I was trying my implementation with varying kernel sizes and strides. I found with a standard (3,3) kernel and unit strides, I’m about 30% off the speed of the cudnn convolution used by Pytorch. However, for non-square kernels and non-unit strides I am roughly equal. In general, though, I’m finding square kernels with unit strides are unbeatable.
I think it heavily depends on the algorithm. My guess is that the unit stride, (3,3) convolution is particularly optimized as it’s simple and very common. I think for that operation, an FFT convolution is faster than im2col so the selector is likely choosing that algorithm, which I am pretty sure can be implemented on a single kernel.
Thanks for the help making sense of this!