Thank you, I will increase the batch size then.
That’s great. I really noticed the data being loaded quickly.
By the way, is there a similar efficient mechanism for loading huge datasets when using your FAISS index library? (with datasets.Dataset.add_faiss_index() and get_nearest_examples)