Clever solution. It worked and sped up my training 4 or 5x.
That being said, there is still a significant speed gap between this and a standard CNN. Is it to be expected that grouped convolutions be slower than standard ones?
Thank you for your help!