It is pretty long so I don’t want to write it all here. But it has worked very robustly prior to adding the labels as an additional input.
It’s the standard loop of training this discriminator, training the generator and can be reduced to:
G.eval()
D.train()

output = G(inputs, labels)

target_pred = D(target, labels)
output_pred = D(output, labels)

loss_D_target = BCE_loss(target_pred, torch.ones_like(target_pred))
loss_D_output = BCE_loss(output_pred, torch.zeros_like(output_pred))
loss_D = (loss_D_target + loss_D_output).mean()
loss_D.backward()

G.train()
D.eval()
optimizer_G.zero_grad()

output = G(inputs, labels)
output_pred = D(output, labels)

loss_G_VGG = VGG_loss(output, target)
loss_G_L1 = L1_loss(output, target)
loss_G_adv = BCE_loss(output_pred, torch.ones_like(output_pred))
loss_G = (loss_G_VGG + loss_G_L1 + loss_G_adv).mean()
loss_G.backward()