More Documents, Same Length: Isolating the Challenge of Multiple
  Documents in RAG
Retrieval-augmented generation (RAG) provides LLMs with relevant documents.
Although previous studies noted that retrieving many documents can degrade
performance, they did not isolate how the quantity of documents affects
performance while controlling for context length. We evaluate various language
models on custom datasets derived from a multi-hop QA task. We keep the context
length and position of relevant information constant while varying the number
of documents, and find that increasing the document count in RAG settings poses
significant challenges for LLMs. Additionally, our results indicate that
processing multiple documents is a separate challenge from handling long
contexts. We also make the datasets and code available:
https://github.com/shaharl6000/MoreDocsSameLen .
67d05aa2348bae81a8ae572e
Shahar Levy
claimed_verified
2025-03-14T08:58:54.727Z
666d93d8357aeb334f1e9a7e
/avatars/c240d1fc938271ce1d4321f5f3fb79d1.svg
Shahar Levy
user
Shahar6000
67d05aa2348bae81a8ae572f
Nir Mazor
claimed_verified
2025-03-19T09:46:45.368Z
6548c950b3a7efb939216dde
/avatars/3a7f48024b45b7154118529cb49ac0ec.svg
Nir Mazor
user
nirmazor
67d05aa2348bae81a8ae5730
Lihi Shalmon
claimed_verified
2025-03-11T16:09:23.471Z
63b433ee7af2e415f25b1a7b
/avatars/0b03f66d263bffd22ed864d1241fe28b.svg
Lihi Shalmon
user
LihiShalmon
67d05aa2348bae81a8ae5731
Michael Hassid
67d05aa2348bae81a8ae5732
Gabriel Stanovsky
More Documents, Same Length: Isolating the Challenge of Multiple
  Documents in RAG
Retrieval-augmented generation (RAG) provides LLMs with relevant documents.
Although previous studies noted that retrieving many documents can degrade
performance, they did not isolate how the quantity of documents affects
performance while controlling for context length. We evaluate various language
models on custom datasets derived from a multi-hop QA task. We keep the context
length and position of relevant information constant while varying the number
of documents, and find that increasing the document count in RAG settings poses
significant challenges for LLMs. Additionally, our results indicate that
processing multiple documents is a separate challenge from handling long
contexts. We also make the datasets and code available:
https://github.com/shaharl6000/MoreDocsSameLen .
Evaluating retrieval-augmented generation with varying document counts while keeping context length constant reveals significant challenges for language models.
retrieval-augmented generation RAG language models multi-hop QA context length document count