So Hard or Semi-hard Negative Mining is the key, without it the net learned to correctly predict only easy examples. Also keep 1:1 ratio between positive and negative examples, or at least weight it in the loss (like param weight in i.e. nn.CrossEntropyLoss).
Also somehow on my machine the loss was calculated wrong with
loss_contrastive = torch.mean((1-label_batch) * torch.pow(euclidean_distance, 2) +
                                          label_batch * torch.pow(torch.clamp(margin - euclidean_distance, min=0.0), 2))

and this
loss_contrastive = torch.mean((1-label_batch) * torch.pow(euclidean_distance, 2) +
                                          (label_batch) * torch.pow(torch.clamp(margin - euclidean_distance, min=0.0), 2))

gave me the loss correctly. I don’t remember if I discovered the core problem of the parenthesis or didn’t have time for that. You can try it yourself - I think that this problem doesn’t occur for everyone.
For me all the three things worked.