I am not sure of the exact reason.
Below are some points that I could think of:

You do not need tow square the euclidean distance, as pairwise distance already gives that.




 dengio:

torch.pow(euclidean_distance.double(), 2)




Check if your dataset is balanced/imbalanced.


I see that you have used only 2-dimensions in the final layer. Try using more (64, 128, 256 etc.,)
I am not sure if reducing 20 dimensions to 2 dimensions is not generalizing well.





 dengio:

nn.Linear(256, 2)




Try normalizing the final descriptor (F.normalize())


Play-around with the margin parameter of siamese (contrastive) loss.
Also, try triplet loss.