Hi, I wrote a basic GAN and wonder if it’s possible to train G and D in the same step.
Many codes I’ve seen use alternating steps to train G and D and I find this inefficient.
The idea is to have two optimizers like this:
opt_G = Adam(G.parameters())
opt_D = Adam(D.parameters())
loss_G = log(1-D(G(z)))
loss_D = log(D(G(z))) + D(real)

Gradients from loss_G depend on both G.parameters() and D.parameters() but opt_G will only update G.parameters().
The problem with using one optimizer is loss_G.backward() will put some values on grad of D.parameters(). This is wrong because loss_G affect D’s parameters.
This is what I am thinking:
opt_G.zero_grad()
loss_G.backward()
opt_G.step()

opt_D.zero_grad()
loss_D.backward()
opt_D.step()



I am not sure G update doesn’t affect D update in this code.


How can I use retain_graph or retain_variables for this? There is no documentation about retain_variables.


If loss_G.backward() can only update grad on G.parameters and abandon D.parameters I could write in one optimizer (Is this retain_variables?) :
opt.zero_grad()
loss_G.backward(G.parameters())
loss_D.backward(D.parameters())
opt.step()