to add to what @shamanez said, i also recently learnt that you can add faiss indices directly in the datasets library: Adding a FAISS or Elastic Search index to a Dataset — datasets 1.6.0 documentation 5