Thank you very much for your suggestions!
I’ve stumbled upon metric learning before but it went over my head how to implement it in my case. I’ve figured it out now, wasn’t too difficult once it clicked with the link you provided and the tests I’ve done show good results. The distance is much much closer now. Sometimes nearly 0 which is a big win!
I’m still struggling to understand what’s really going on and how this works. From my understanding every layer has weights that can be tuned, in fine-tuning you freeze most of the pre-trained weights. I’m not freezing any right now and I wonder if that’s the best thing to do.
From my intuition I’d freeze every weight but the last one I use, the avg_pool but this one and those before don’t have much weights. I fear I skew the weights too much with my limited data set.
Any suggestions on this or do you think it’s alright?
For reference, I build the model like this:

model = tf.keras.applications.EfficientNetB0(include_top=False, weights=‘imagenet’, pooling=‘avg’)
embeddings = tf.nn.l2_normalize(model.output, axis=-1)
metric_model = EmbeddingModel(model.input, embeddings)
model.summary()