This is actually a topic in computer vision research called “fine-grained image classification.” You may take a look at the literature and pick a most relevant one. I personally recall some methods like “tripplet loss.” You may want to take a look at it.