Hi all,
I’m writing a GAN for image translation.
I’m using a kinda unet as a generator and a resent (from the torch hub) as a discriminator (modified to have 1 class, sigmoid).
I train the models by alternating minibatches (discriminator, generator, discriminator, and so on).
The first discriminator minibatch runs, the first generator minibatch runs, the second discriminator minibatch runs too but when it switches to the second generator again I get this:

Traceback (most recent call last):
File “/home/paolo/cycleGAN/net_CT/35/…/…/cycleGAN/train.py”, line 704, in 
run_training(models, trainCases, epoch, lp, max_image_shape, device)
File “/home/paolo/cycleGAN/net_CT/35/…/…/cycleGAN/train.py”, line 442, in run_training
loss_G.backward()
File “/home/paolo/cycleGAN/venv_nightly/lib/python3.9/site-packages/torch/_tensor.py”, line 255, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File “/home/paolo/cycleGAN/venv_nightly/lib/python3.9/site-packages/torch/autograd/init.py”, line 147, in backward
Variable._execution_engine.run_backward(
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

My environment is Arch Linux box, nvcc 11.2, python 3.9.1, cudnn 8.0.5 (but it is installed also cudnn 6.0).
I get the same error both if I install the nightly or the stable (pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html 1).
I think the models and the metrics are ok since a few minibatches run.
Thank you.