Hi Jung & HF Community.
I am implementing a RAG process,… with a daily update.
I can easily merge the dataset objects using datasets.concatenate_datasets()
but I have two questions:

I cannot merge the indices… even if i .load_faiss_index() to each part the concat object has no index
Is this the best way to search a large corpus or would it be best to load each dataset into a seperate node and scan across a cluster?

I am following transformers/use_own_knowledge_dataset.py at master · huggingface/transformers · GitHub 1, creating a new folder for each daily dataset.