I have a siamese network which I am training on positive and negative samples and minimizing the cross entropy loss between the outputs(pos/neg) with their corresponding targets. However, the test accuracy ALWAYS seems 50% (i.e 50% of the test sets classified as correct, presumably the positive samples) and I am not sure how to trouble shoot this issue 
I was wondering of the pytorch community had any tips for me to trouble shoot this as I seem lost!