mdlockyer:

Regarding setting requires_grad to False on D while back propagating G’s loss, I have been meaning to implement that same thing but never got around to it. It seems like the logical approach, as it is just wasting compute time calculating gradients for D when they are going to be discarded.


I think that doing exactly that would make it work with a single process group. Because you no longer race the allreduce calls from the two models. Also, I think you could put the discriminator in eval mode when doing this, which side steps some of the synchronization code paths in DistributedDataParallel.