The repository depends on Deformable-Convolution-V2-PyTorch, which seems to have been written ~3 years ago. Are you also seeing a slowdown without these custom layers or did you profile the model to see which operations are the bottleneck?