I meant performance as model accuracy. I use mAP as performance measure. The goal is to conclude on the similarity of inputs, that is why I have an early return of feature vector in the if not self.training: return branch_outs. During training it can be looked at as a simple classification problem
Also loss does not go down during training: 
The orange/blue line is a good performing resnet50, the red is my own rebuilt model.
Also note that every other part of my code is the same, such as loss, optimizer, dataloader, data, etc. I investigated thoroughly and im pretty sure the problem is the model file.