I am creating positive and negative pairs for Contrastive loss calculation in Siamese network. For positive pairs I am taking two similar images. However for creating a negative pair I am randomly pairing two different images. I am still working on the model architecture. Every time I train a new model a random combination of dissimilar pair is picked. This causes different accuracy and loss function. Am I right to pick dissimilar pairs randomly or should I create fixed similar and dissimilar pairs for training and testing?
Also what should be the ratio for similar and dissimilar pairs for every image?