I am new to PyTorch, and attempting to run some Mask R-CNN code using PyTorch on AWS Ubuntu 16.04 with a Tesla K80 GPU. I am using python 3.6 and latest PyTorch code compiled from source. I am able to run PyTorch examples and some basic test code on GPU. However, when running a custom extension, i get this error:
RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /home/ubuntu/pytorch/aten/src/THC/generated/…/generic/THCTensorMathPointwise.cu:386
If it helps, I am running a non-maximal suppression code from here: https://github.com/ruotianluo/pytorch-faster-rcnn/tree/master/lib/nms/ 37 . I compiled the code using nvcc -c -o nms_kernel.cu.o nms_kernel.cu -x cu -Xcompiler -fPIC -arch=sm_37  and then built using the build.py which all went fine as far as I can tell.
Full trace below:
THCudaCheck FAIL file=/home/ubuntu/pytorch/aten/src/THC/generated/…/generic/THCTensorMathPointwise.cu line=386 error=48 : no kernel image is available for execution on the device
Traceback (most recent call last):
File “model/mask_rcnn_lib/box.py”, line 335, in 
run_check_nms()
File “model/mask_rcnn_lib/box.py”, line 324, in run_check_nms
keep = torch_nms(rois, 0.5)
File “/home/ubuntu/DSB_2018/HCK_MRCNN/model/mask_rcnn_lib/nms/torch_nms/init.py”, line 34, in torch_nms
areas = (x2 - x1 + 1) * (y2 - y1 + 1)
RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /home/ubuntu/pytorch/aten/src/THC/generated/…/generic/THCTensorMathPointwise.cu:386
Thanks and apologies if this is the wrong place to post this.