Using nvidia tensorrt 20.11 image and pytorch 1.7.1,
Running this
find . -type f -exec sed -i 's/THCState_getCurrentStream(state)/c10::cuda::getCurrentCUDAStream()/g' {} \;

And replace external/DCNv2/src/cuda/dcn_v2_cuda.cu file by this version DCNv2/dcn_v2_cuda.cu at c916fdf2ed69ebebc43cbe8a2cb5e7ce0dd71132 · lbin/DCNv2 · GitHub 37
I manage to get a successful build, but I have the same issue mentionned above when importing dcn_v2 :
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "yolact/external/DCNv2/dcn_v2.py", line 13, in <module>
    import _ext as _backend
ImportError: yolact/external/DCNv2/_ext.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at4cuda4blas4gemvIfEEvcllT_PKS3_lS5_lS3_PS3_l