I have a question regarding how to implement the test phase in the system described on the image

forum_flowchart458×505 2.33 KB

I have trained both N1 and N2 to transform an input image (Im) wrt to Im’s class. If Im belongs to class 1 (C1) then I updated N1. If it belongs to C2 I update C2. At the same time I update the classifier to reckon the transformation which is inherent to each class.
Now, that the system is trained I want to predict new incoming samples. As I do not have any label that tells me which network (N1 or N2) has to transform Im, then I need to feed it to both and therefore I will have two transformed images T1 and T2. My classifier needs to assign two labels now during test but I need just one. In theory, if the input image belongs to class 2 then T1 is an image that the classifier should classify with low confidence whereas T2 should have a higher confidence and viceversa. My problem is that I dont know how to implement this idea. I have tried too many things. The output of my classifier is a vector of two values for each image. Therefore I will have a pair of vectors (C1_t1,C2_t1) and (C1_t2, C2_t2) indicating the probabilities of each transformed image T1, T2 of belonging to classes C1,C2. I need to find a way from those values to output the final class.