Just for the sake of completeness: The profiling times in my first post are not correct, but I cannot edit the post anymore. Here are the correct ones:
# Output of deform_conv_speed.py

bs: 01, in-channels: 512, groups: 1
Time normal conv:                0.37 ms
Time torchvision deform conv:    0.37 ms
Time mmdet deform conv:          0.36 ms
----------------------------------------
bs: 01, in-channels: 512, groups: 512
Time normal conv:                0.19 ms
Time torchvision deform conv:   15.23 ms
Time mmdet deform conv:         13.67 ms
----------------------------------------
bs: 16, in-channels: 512, groups: 1
Time normal conv:                1.15 ms
Time torchvision deform conv:    2.26 ms
Time mmdet deform conv:          2.19 ms
----------------------------------------
bs: 16, in-channels: 512, groups: 512
Time normal conv:                0.32 ms
Time torchvision deform conv:   15.06 ms
Time mmdet deform conv:         13.76 ms
----------------------------------------