Thanks, AlbanD,
after using CUDA_LAUNCH_BLOCKING=1 , I got:
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=77 : an illegal memory access was encountered
Traceback (most recent call last):
File “test.py”, line 103, in 
train(epoch)
File “test.py”, line 86, in train
loss.backward()
File “/usr/local/lib/python2.7/dist-packages/torch/tensor.py”, line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “/usr/local/lib/python2.7/dist-packages/torch/autograd/init.py”, line 90, in backward
allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /pytorch/aten/src/THC/THCGeneral.cpp:663
BTW, loss definition is torch.nn.functional.nll_loss() and the problem happens after several backward training steps.