Hi fellows!
I’m implementing a model that is inspired in the music translation network by Facebook.
Essentially, we have an encoder network (take input belonging to some class and output latents) and a classifier that tries to predict the class of the input solely based on its latents. The minimax game here is for the encoder to seek for class-independent latents and the classifier to be able to correctly classify them.
The original implementation avoids using retain_graph by first computing the latents and optimizing the classifier with a classification loss, and then recomputing the latents (with exactly the same input if the data augmentation option is disabled) and optimizing the encoder with the negative classification loss (the same loss function but recomputed to use the updated parameters of the classifier).
My question is why do we have to use retain_graph if the optimizers are on disjoint sets of parameters? And, is using retain_graph the same as recomputing the latents in this case?
Thanks for your help!