Ok, the whole thing about training accuracy being “good” and evaluation accuracy being “bad” is to do with labels. It’s like training a binary classifier where the first half of a batch is label 0 and the second half of the batch is label 1. You will find that training accuracy goes up, but evaluation accuracy is almost always close to 0. It’s the same kind of behavior here. You need to randomize your batch and therefore your labels. By using the loss L → 2*L(A2,B2) - L(A1,B2) - L(A2,B1), I’m passing all “good” labels in one batch and all “bad” labels in another. So the workaround is to have another loss function where you can minimize the following L = L1(randperm(cat(A1,A2)), B2) + L2(A2,randperm(cat(B1,B2)). Something like that. You need to randomize the batch when concatenating A1 and A2, or B1 and B2. You need to appropriately define L1 and L2 for your problem.
So having done all of that, you can train (A2,B2) to be maximally different to (A1,B1). And likewise you can train (A3,B3) to be maximally different to (A2,B2). But if you only ever load 4 models, 2 of which are frozen and the other 2 are training, there is no guarantee that (A3,B3) is maximally different to (A1,B1). Your models could be theoretically oscillating between two local minima. And there is a limit to how many models you can load at once. In theory you could load all of (A1,B1), (A2,B2) and (A3,B3) and make (A3,B3) different to the last two pairs, but not everyone has 24GB of GPU memory.
So, …, it would be nice if there was an easy and computationally friendly way to guarantee different models every time you trained them. What is it?