Let me see if I can produce a minimal example, and I’ll get back to you. Some of this is intertwined with other stuff I’m working on. I’ll try to throw together a mini-profiler.
In theory, this should lend itself nicely to the im2col convolution variant at the very least. Instead of parallelizing the copy over C x H x W, it would parallelize it over N x C x H x W (as NHW in this case is roughly the same as HW in a Normal spatial convolution over larger images) and have some extra checks in the double for-loop over the filter data to make sure that it’s not copying from the wrong “patch”.