Yeah, you are right. Since you call d_loss.backward right after it, you need to retain the gradients.
Well, it seems in your setup you have one generator and two discriminators, one for frames and the other for sequences.
In the vanilla DCGAN training, you start by upgrading the discriminator.
Here the gradients for D using real samples are calculated. Note that no update step is performed yet.
In this section 2 we use G to generate a fake sample, detach it so that no gradients will be calculated in G, and feed it to D to train it with this fake sample. After the backward call was called, the optimizer updated the parameters of D.
Finally in these lines of code we update G using the fake sample we created before, but this time without detaching it when passing to D, and using fake labels.
I’m not sure, how frame/sequence_discriminator of the loss functions are implemented, but you should take care that the right gradients are calculated for each model.
However, I think these points are unrelated to your OOM issue.
So back to the original question: if you use -dis_loss() then everything works, but if you split it to frame_loss and sequence_loss you get the OOM error?
Could you share the implementation of dis_loss?