ZimoNitrome:

Not switching to train/eval mode, it would seem that doing backprop on D(G(noise)) would train both D and G simultaneously. Is this incorrect?


No, since model.eval() and model.train() do not change the gradient calculation or the backpropagation in general. Instead the behavior of certain layers is changed. E.g. dropout layers are disabled during eval() and batchnorm layers will use their internal running stats to normalize the input activation.



 ZimoNitrome:

Furthermore, trying to switch between eval and train like I suggested above seems to destabilize the GAN, where D gets 100% accuracy for real samples but 0% accuracy for fake samples.


I think this is the reason why this particular tutorial keeps both models in train() mode.



 ZimoNitrome:

Maybe freezing D and G with param.requires_grad = False is better? Trying that out seems to make little difference if any to the original training scheme though.


I don’t think this would make any difference, since the output of the generator is already detached when the discriminator is trained as seen here.