You have a typo in your profiling script and are using the mmdetection implementation twice instead of the torchvision one. 
That being said, you can see the torchvision CUDA implementation here 26, which uses an im2col approach and applies the bilinear interpolation.
Note that “normal” convs are heavily accelerated for “typical” shapes in cudnn and an im2col approach would not yield the same performance.
The paper mentions

Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from the target tasks, without additional supervision.

which seems to learn the offsets and thus selecting a fast kernel for a specific workload seems to not work out of the box.