Thanks @ptrblck. You were right, instead of using a traditional sample wise train-test split, I reserved a few set of classes for validation from my training set, and for validation sampled pairs from this validation set and computed the pair-wise matching accuracy.