Valerio_Biscione:

emb_all = model.image_embedding(x)
emb_A = emb_all[:k]
emb_B = emb_all[k:]



Just a wild guess. Do you use batchnorm in your network?
If so, if A belongs to a different distribution of images than B, then batchnorm statistics may be widely different and they would have a tough time learning statistics, when you pass them individually I guess. In the case that you pass them together, they might get normalized properly.