I meet this error as following, is there some actions is not differentiable or in-place operation cause this error. I want to know how to debug this error.
Traceback (most recent call last):
File “training/train_super_pixel_net.py”, line 370, in 
main(args)
File “training/train_super_pixel_net.py”, line 103, in main
avg_loss = train(args, model, optimizer, epoch, trainloader, scheduler, best_iou, loss_fn)
File “training/train_super_pixel_net.py”, line 151, in train
loss.backward()
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/tensor.py”, line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/autograd/init.py”, line 90, in backward
allow_unreachable=True)  # allow_unreachable flag
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/autograd/function.py”, line 76, in apply
return self._forward_cls.backward(self, *args)
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/_functions.py”, line 30, in backward
return (None,) + ReduceAddCoalesced.apply(ctx.input_device, ctx.num_inputs, *grad_outputs)
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/_functions.py”, line 41, in forward
return comm.reduce_add_coalesced(grads, destination)
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/cuda/comm.py”, line 119, in reduce_add_coalesced
flat_tensors = [_flatten_dense_tensors(chunk) for chunk in chunks]
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/cuda/comm.py”, line 119, in 
flat_tensors = [_flatten_dense_tensors(chunk) for chunk in chunks]
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/_utils.py”, line 144, in _flatten_dense_tensors
flat = torch.cat([t.contiguous().view(-1) for t in tensors], dim=0)
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /pytorch/aten/src/THC/THCCachingHostAllocator.cpp:271