I did not implement a validation step because I came up with the same question. I could use the GT label during validation, because if you do that you know which network N1 or N2 the input should be fed through and that’s not how it is supposed to be evaluated. Therefore, I have just trained the model.

Once approach would be to use a small hold-out set to train another classifier on top of your current model to use the 4 predictions and output only the two classes.

What do you mean by that? If I trained a classifier in top of both N1 and N2 to predict the label of the input am I leveraging the fact of N1 and N2 transforming the input differently with regard to the class it belongs to? I want to output only one class. So if input belongs to C1 the classifier should be used to see T1-like images and not T2-like as in training never happened a C1 input to be fed into N2. But actually it seems that my classifier did not learn about class features of the image but only about the transformation. I don’t know if my thoughts are correct.