First of all, thank you very much for your answer.
Based on your answer, the problem is most probably because of the wrong settings that the Cuda codes are compiled. The command to build Cuda codes was
nvcc -c -o nms_kernel.cu.o nms_kernel.cu -x cu -Xcompiler -fPIC -arch=**sm_60**
and I will change it to
nvcc -c -o nms_kernel.cu.o nms_kernel.cu -x cu -Xcompiler -fPIC -arch=**sm_37**
I hope it will work this time with PyTorch 0.4.1.post2 in Google Colab.
Moreover, there should be some misleading information in Google Colab about Tesla K80. Below picture was the output of the command of !nvidia-smi

Finally, I want to ask two more things about the installation from source process. If I want to build PyTorch from the source, how to compile it with a specific CUDA version? Or is it automatically built with the CUDA installed in the machine?
And is there any compatibility table that shows PyTorch versions and their compatible CUDA versions? I am asking this because I have RTX 2080 Super card in my local machine and I may have to run PyTorch 0.4 on my local machine in near future. RTX2080 super has compute capability of 7.5.
Thank you again your kind answer.