I have 2 generators. Both are competing against each other in typical GAN fashion.
The 1st generator acts like an auto-encoder and the 2nd generator outputs a different result.
I want to introduce a 3rd network, i.e. a classifier, that classifies the input and output of the 1st generator into 1 of 2 classes: encoded and non-encoded.
When training, all three losses (1st, 2nd generators and classifier losses) decrease nicely, but when evaluating, the classifier loss is high (not the generators’). Now, you want the encoded and non-encoded tensors to be similar but I would expect the classifier to still perform well during evaluation.
Is there any reason why 2 networks (2nd generator and classifier) competing against 1 network (1st generator) wouldn’t work?
Details:
The classifier is an inceptionresnetv2 classifier taken from cadene/pretrainedmodels.
The classifier loss is CrossEntropy (implicit logsoftmax activation function).
All three losses are summed together to give a master loss. I then call loss.backward(), which should back-propagate gradients back through all three networks.