@ptrblck
Thanks for the response, yes I agree that it’s weird that they computed fake_img and fake_out after  the loss and optimizerG.step().
I tried 2 different changes that seems to work and while the training now works, I’m not sure if my changes afftect the overall intended training loop and model’s accuracy:
(Changes can be seen here: https://github.com/YoadTew/SRGAN/blob/master/train.py)

I changed the order of fake_img and fake_out, and it fixed the problem.

            netG.zero_grad()

            fake_img = netG(z)
            fake_out = netD(fake_img).mean()

            g_loss = generator_criterion(fake_out, fake_img, real_img)

            g_loss.backward()
            optimizerG.step()


I removed retain_grph=True in d_loss.backward(), and sent fake_img.detach() to create new fake_out for netD.

The thing is, I still dont know the reason for the error in the first place, and if my changes will change the intended flow of the training script.