In the context of object tracking, until recently, the best feature extractor for Siamese Networks was AlexNet. This paper 8 sheds light on the reasons why standard ResNets worsen feature extractor performance, and they propose a solution to this problem.
I’m not sure if it will solve your case, but it is worth taking a closer look.