I think I found a rather ugly hack to increase the performance of my network.
If I understood the concept correctly, a grouped (depthwise) convolution is essentially a normal convolution where only the i-th channel of the i-th kernel contains non zero weights. That means I can train my network using grouped convolutions (groups = in_channels) and during inference I use full convolutions (groups = 1) but I initialize the kernels with 0 and insert the previously trained kernels at the i-th channel of the i-th kernel. I think the code snippet below illustrates better what I’m doing.
# model.deform_conv.weight.shape = [512, 512, 3, 3]
# pretrained_kernels.shape = [512, 1, 3, 3]

for i in range(model.deform_conv.weight.shape[0]):
    model.deform_conv.weight[i, i, :, :] = pretrained_kernels[i, 0, :, :]

I modified my code accordingly and can confirm that the results are exactly the same as with grouped convolutions but the inference time is decreased by a factor 8.
Now I’m wondering if this special case of grouped convolutions is really that hard to implement? Couldn’t the “hack” that I explained above also be implemented on CUDA level?
@ptrblck Maybe you can tell me if this is feasible or not? Thanks a lot!