Thanks for your reply!
Regarding to your answer, the .backward() doesn’t go through the decoder in the case of optimizing the classifier since its weights are not used at any point of the computation graph.
My confusion was more about that you have to explicitely detach the input variable even if it’s not used in the optimizer. For me it’s then more natural to compute the classifier loss detaching the latents and then without need of recomputing the latents, compute the same loss to optimize the encoder.
The simplified code of the music translation network looks then something like this. And you don’t need to recompute the latents.
# Optimize D - discriminator right
z = self.encoder(x)
z_logits = self.discriminator(z.detach())
discriminator_right = F.cross_entropy(z_logits, torch.tensor([dset_num] * x.size(0)).long()).mean()

self.d_optimizer.zero_grad()
discriminator_right.backward()
self.d_optimizer.step()

# optimize G - reconstructs well, discriminator wrong
y = self.decoder(x, z)
z_logits = self.discriminator(z)
discriminator_wrong = - F.cross_entropy(z_logits, torch.tensor([dset_num] * x.size(0)).long()).mean()