Hi ! that’s a mistake in the eval_rag.py parameters choices. As specified in the rag configuration (see documentation), one can choose between ‘legacy’, ‘exact’ and ‘compressed’. The legacy index is the original index used for RAG/DPR while the other two use the datasets library indexing implementation.
link to the PR that fixes the eval_rag.py parameter description: https://github.com/huggingface/transformers/pull/8730 6