So I went and rolled my own “Patch Convolution,” reimplementing im2col and col2im with a 5-D tensor (B x C x P x H x W) and launch it with C * P * H * W threads. It’s definitely slower than cudnn would be (my CUDA skills are naive at best), but it runs the same network as described above in ~1.6s per batch. I know it’s not really possible to really demonstrate the difference without that profiler I promised, but it seems to me that this result suggests that the “optimal kernel” for my problem just might not be in the cudnn library, as you posited.
In the meantime, I can live with my 5x slowdown for now…
On another note, this operation is totally something that PyTorch should support though. It’s a fairly trivial change from the standard im2col for conv2D. The only major differences are in how we stride and index. I’m happy to share my baseline implementation, but I am sure the PyTorch devs can do a better job than me.