Thanks @ptrblck for having a look at my script and spotting the typo! I updated both the profiling output and the script.
I already feared that it would not be straight forward to optimize for the grouped convs case. A possible alternative for me would be a applying the same 3x3 kernel to all channels. I read that something like this can be done by merging batch and channel dimensions together, performing the conv and then splitting them again. However, in my case I have different offsets for each image in a batch, so I cannot perform the exact same convolution over all channels in a batch (even though the kernel would be the same).
Do you have any idea how to solve this?