Hi, everyone.
I try to run a cleaner version of pytorch CycleGan implementation in this link:
https://github.com/aitorzip/PyTorch-CycleGAN 4
However, during training procedure, I met a error in backward propagation.

Traceback (most recent call last):
File “D:/wangyesFile/pytorchlearning/PyTorch-CycleGAN-master - Copy/train.py”, line 165, in
loss_G.backward()
File “D:\Python36\lib\site-packages\torch\tensor.py”, line 102, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “D:\Python36\lib\site-packages\torch\autograd_ init _.py”, line 90, in backward
allow_unreachable=True) # allow_unreachable
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at c:\a\w\1\s\tmp_conda_3.6_091443\conda\conda-bld\pytorch_1544087948354\work\aten\src\thc\generic/THCStorage.cpp:39
I have searched this problem with google and found some similar errors when using nn.Conv3d and nn.ConvTranspose3d.
The code in cyclegan utilizes nn.ConvTranspose2d in generator. I wonder if there is a relationship between this error and nn.ConvTranspose2d.

It is a weird problem because it doesn’t happen in every training step, and has no rule to follow.
My environment is Win10, pytorch1.0, cuda9.2 and cudnn7.4.
I add
os.environ[‘CUDA_LAUNCH_BLOCKING’] = ‘1’
in my script. The error turned to this:

File “D:/wangyesFile/pytorchlearning/PyTorch-CycleGAN-master - Copy/train.py”, line 166, in
loss_G.backward()
File “D:\Python36\lib\site-packages\torch\tensor.py”, line 102, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “D:\Python36\lib\site-packages\torch\autograd_ init _.py”, line 90, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

But I still have no idea about why this happen and how to fix it.
Thanks for any suggestions.