Given that Siamese networks can work for one-shot learning (such as image recognition (Koch, Zemel & Salakhutdinov (2015) 3, Jadon (2020) 2), maybe you could try adapting these Siamese net examples from the Keras community @fchollet:

Image similarity estimation using a Siamese Network with a triplet loss 7
Image similarity estimation using a Siamese Network with a contrastive loss 4

Perhaps, @Sayak_Paul knows more about this.