terbed:

Here we should also zero_grad() the discriminator network or the situation is the same here and it is not a problem that gradients are accumulated in the discriminator?


I wouldn’t say “it’s not a problem”, as it depends on your use case.
If you want to update a specific model, e.g. G in your use case, in the current code block, then you could either let PyTorch calculate the gradients for C and D and zero them out before updating C and D in their “update blocks” or alternatively you could set their requires_grad attribute of all parameters temporarily to False.
Both approaches would work and the critical step is to make sure the optimizer uses only “valid” gradients to update the corresponding model.