Hi
I’m working on a project where I use deformable convolutions to shift features in the spatial dimension. Since I only want to shift them spatially it seemed logical to use depthwise convolutions which can be done by using groups = in_channels = out_channels as far as I understand.
Unfortunately the deformable convolutions implemented in torchvision.ops.DeformConv2d as well as the ones implemented in mmdetection 7 are very slow when using groups = in_channels (see time measurements below).
I’d like to avoid using full convolution kernel (3x3xin_channels) because I don’t want to increase the capacity of my network too much. Is there a way to a) either speed up the deformable convolutions with group option, or b) use the same 3x3 kernel for each of the input channels?
Thanks in advance =)
Script used to measure times: https://gist.github.com/MauroPfister/6643169d1b0d9b656277dbaaa8548745 25
# Output of deform_conv_speed.py

bs: 01, in-channels: 512, groups: 1
Time normal conv:                0.35 ms
Time torchvision deform conv:    0.37 ms
Time mmdet deform conv:          0.38 ms
----------------------------------------
bs: 01, in-channels: 512, groups: 1
Time normal conv:                0.35 ms
Time torchvision deform conv:    0.34 ms
Time mmdet deform conv:          0.31 ms
----------------------------------------
bs: 16, in-channels: 512, groups: 512
Time normal conv:                0.31 ms
Time torchvision deform conv:   15.46 ms
Time mmdet deform conv:         14.05 ms
----------------------------------------
bs: 16, in-channels: 512, groups: 512
Time normal conv:                0.35 ms
Time torchvision deform conv:   15.46 ms
Time mmdet deform conv:         14.06 ms
----------------------------------------