Well if Tensorflow/keras isn’t a necessity you can try CLIP model by OpenAI. It is quite robust and has good generalizability. You can pre calculate all the images embeddings and calculate cosine similarity with your query image. For huge number of images at large scale you can use Approximate Nearest Neighbor with Hnswlib:
  
      

      github.com
  

  
    

nmslib/hnswlib 4


  Header-only C++/python library for fast approximate nearest neighbors

  

  
    
    
  

  


or Faiss:
  
      

      github.com
  

  
    

facebookresearch/faiss 4


  A library for efficient similarity search and clustering of dense vectors.

  

  
    
    
  

  


or Annoy:
  
      

      github.com
  

  
    

spotify/annoy 4


  Approximate Nearest Neighbors in C++/Python optimized for memory usage and loading/saving to disk

  

  
    
    
  

  


CLIP is usually used by encoding the text to same embedding domain as images, but it should work within images itself too since they are in same domain.