First of all, thank you for your answers, I made it to run my code with pytorch 0.4 with cuda 9 in Google Colab. But there was a bug in my code in pytorch 0.4  and I downgraded it to pytorch 0.3 and it worked.
However, in my local setup, since RTX2080s required Cuda 10 minimum, I couldnt find any binary of pytorch 0.3 compiled with cuda 10 for linux in this link https://download.pytorch.org/whl/cu100/torch_stable.html.
So, most probably, there are two options for me now.

I will try to compile PyTorch 0.3 from the source with Cuda 10
I will try to rewrite nms and roi align extensions written in c and cuda to make the code work with the last version of PyTorch(1.9).

I am not sure which one will be difficult to do. Because without rewriting, I hit this error ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead with last version of pytorch.