Have you looked at what “im2col” does? It’s non trivial.
You can do it, but it’s going to be tricky and not really useful as you are in the worst possible case: many small ops that run on the GPU. What will cost the most for you is actually going to run the python code and send jobs to the GPU. To get a small gain, you would need to use multiple processes which are not supported by autograd out of the box and so you will need to unpack/pack Tensors when sending them to make sure it will work.
So yes you can parallelize the loop while maintaining gradients, but it’s going to be hard and I don’t think you’re going to see any big improvement by doing so.