Actually, it was the problem when I compiled some file written in cuda.
Please check the followings:

Pytorch is compiled in the same version with system cuda version
Choose the true architecture code for your GPUs. For example, in this line https://github.com/rowanz/neural-motifs/blob/d05a251b705cedaa51599bf2906cfa4653b7a761/lib/fpn/nms/src/cuda/Makefile#L2 86, the GPU architecture is 61 for Titan Xp. You must change it into 52 for Geforce GTX Titan, or 75 for Titan V.
Hope this helps.