I was getting RuntimeError: CUDNN_STATUS_MAPPING_ERROR while trying to run
F.conv2d(x, conv_reduce_w, stride=1, padding=0)
The above line of code runs successfully if I am using CPU only.
I then added torch.cuda.synchronize() before executing this line of code, then I get
torch.cuda.synchronize() File "/u/pahujava/anaconda3/envs/venv/lib/python3.6/site-packages/torch/cuda/__init__.py", line 314, in synchronize return torch._C._cuda_synchronize() RuntimeError: cuda runtime error (59) : device-side assert triggered at torch/csrc/cuda/Module.cpp:267
I am running my script with CUDA_LAUNCH_BLOCKING=1 and torch.backends.cudnn.benchmark = False
though no change in error is observed even when torch.backends.cudnn.benchmark = True
I tried to make both input tensors contiguous, still no change, any help is appreciated.
FYI, I have checked the tensor dimensions, they are compatible. The dimensions of x and conv_reduce_w are torch.Size([10, 1024, 14, 14]) and torch.Size([128, 1024, 1, 1]) respectively.
Pytorch version: 0.3.1.post2
CUDA version: 9.1