I have a pair of models, A1 and B1 that I train on some data on date 1. How can i train a new pair A2 and B2 on date 2, such that the combinations (A1,B2) and (A2,B1) are guaranteed to fail?
Currently if i have some loss L(A,B) which i minimize, i update it as follows:
L → 2*L(A2,B2) - L(A1,B2) - L(A2,B1)
So when training A2 and B2, i load A1 and B1, freeze their weights and i ensure that their parameters are not added to the optimizer. So i have 4 models loaded in total, but only 2 are training.
That seems to work during training, but not during evaluation. Like training accuracy is fine but evaluation accuracy goes down the toilet. Is there a known way to do this?