For the moment I use this approach only during inference. That is, I train with grouped convolutions (which does not seem to be much slower) and modify the network afterwards for faster inference.
I did not think about using the same approach during training, but it is a great idea! Do you think it would be sufficient to initialize theses “zero-padded” kernels once in the beginning because all the zero weights would generate a zero gradient?
If that does not work, at which point would I have to zero out the gradients? Right before taking the optimizer step?