how we can replace/swap the Wikipedia data with our custom data for knowledge retrieval in the RAG  model and  the format of the retrieval data.  
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:
I am also playing with RAG model and I am trying to understand how to replace the Wikipedia data with my custom data. From transformes/model-cards/facebook/rag-sequence-nq I see that the train dataset is wiki_dpr.  So I loaded it with the following
`
from datasets import load_dataset
dataset = load_dataset("wiki_dpr")
`
The dataset is loaded with arrow into RAM(it's prety big, 75 GB). I was wandering, the custom dataset must have the same format as `wiki_dpr` ? If you can help  with a tutorial on how to replace wiki dataset with a custom one, it will be very helpful. Thank you :D<|||||>I think @lhoestq is working on this right now? <|||||>Yes indeed. I'll create the PR later today to allow users to use their own data. I'll also add code examples<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hi, Sorry to bump a closed issue but it's the first one in a google search for Rag (or DPR) and huggingface. 

Could we please have links here for the code exemples or informations about custom datas ? 

Thanks a lot :smiley: <|||||>Hi !
Sure. You can find more information in [the RAG examples readme](https://github.com/huggingface/transformers/tree/master/examples/research_projects/rag#use-your-own-knowledge-source). Also feel free to take a look at [the code that shows how to do it step by step](https://github.com/huggingface/transformers/blob/master/examples/research_projects/rag/use_own_knowledge_dataset.py).<|||||>from this link using snippet of code to download the Wikipedia data.(https://huggingface.co/datasets/wiki_dpr/blob/main/wiki_dpr.py)

this is the command to download all 50 batches of data : !wget https://dl.fbaipublicfiles.com/dpr/data/wiki_encoded/single/nq/wiki_passages_{1..50}.

Then based on your system config merge the batches and go further.

