Thanks a lot, @ptrblck. This is exactly what I wanted.
Just did a quick comparison and it seems that the grouped convolution is a lot faster than for loop.
A single forward pass used to take 0.01 seconds now it’s 0.0002 seconds.