title: Leave No Document Behind: Benchmarking Long-Context LLMs with Extended
    Multi-Doc QA
summary: Long-context modeling capabilities have garnered widespread attention,
  leading to the emergence of Large Language Models (LLMs) with ultra-context
  windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually
  catching up. However, existing benchmarks employ irrelevant noise texts to
  artificially extend the length of test cases, diverging from the real-world
  scenarios of long-context applications. To bridge this gap, we propose a novel
  long-context benchmark, Loong, aligning with realistic scenarios through
  extended multi-document question answering (QA). Unlike typical document QA, in
  Loong's test cases, each document is relevant to the final answer, ignoring any
  document will lead to the failure of the answer. Furthermore, Loong introduces
  four types of tasks with a range of context lengths: Spotlight Locating,
  Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic
  and comprehensive evaluation of long-context understanding. Extensive
  experiments indicate that existing long-context language models still exhibit
  considerable potential for enhancement. Retrieval augmented generation (RAG)
  achieves poor performance, demonstrating that Loong can reliably assess the
  model's long-context modeling capabilities.
paper_authors._id: 667be60309cb8b5498ec90f8
paper_authors.name: Minzheng Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T10:18:23.390Z
paper_authors._id: 64bcc373ef8c0e42bf16acc5
paper_authors.avatarUrl: /avatars/873308203d28115ae1a9e4d0e26508f4.svg
paper_authors.fullname: mz.w
paper_authors.type: user
paper_authors.user: iiiiwis
paper_authors._id: 667be60309cb8b5498ec90f9
paper_authors.name: Longze Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:22:16.572Z
paper_authors._id: 64c7b4d1c547ed5243c07b6c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c7b4d1c547ed5243c07b6c/h96CLBj6dcm01soK2UJzr.jpeg
paper_authors.fullname: Longze Chen
paper_authors.type: user
paper_authors.user: lzchen2001
paper_authors._id: 667be60309cb8b5498ec90fa
paper_authors.name: Cheng Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:22:26.160Z
paper_authors._id: 665813e398dd5097821b7df0
paper_authors.avatarUrl: /avatars/baddbb04c38a6143224031f822e51f56.svg
paper_authors.fullname: cheng-fu
paper_authors.type: user
paper_authors.user: cfcloud
paper_authors._id: 667be60309cb8b5498ec90fb
paper_authors.name: Shengyi Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:22:37.397Z
paper_authors._id: 64a69c1a9704e9888c1da827
paper_authors.avatarUrl: /avatars/a3b01e482de7b42c9b4de9be347fe187.svg
paper_authors.fullname: LiaoShengyi
paper_authors.type: user
paper_authors.user: LsyLsyLsyyy
paper_authors._id: 667be60309cb8b5498ec90fc
paper_authors.name: Xinghua Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:22:48.990Z
paper_authors._id: 656426e7ec7e2398990a2d34
paper_authors.avatarUrl: /avatars/e84aed6b55b0554ad9581ae3c138a16a.svg
paper_authors.fullname: AIRobotZ
paper_authors.type: user
paper_authors.user: AIRobotZ
paper_authors._id: 667be60309cb8b5498ec90fd
paper_authors.name: Bingli Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:22:59.506Z
paper_authors._id: 64b75f7f00bac1088ceafc06
paper_authors.avatarUrl: /avatars/8ce3ce9901d76ec96574769bdb11b8be.svg
paper_authors.fullname: wubingli
paper_authors.type: user
paper_authors.user: wubingli
paper_authors._id: 667be60309cb8b5498ec90fe
paper_authors.name: Haiyang Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:08:21.206Z
paper_authors._id: 64a4ceda9a90f701134189b7
paper_authors.avatarUrl: /avatars/859a189c5d2ae2fcb9aa2d79104fbfe7.svg
paper_authors.fullname: Haiyang Yu
paper_authors.type: user
paper_authors.user: yhycai
paper_authors._id: 667be60309cb8b5498ec90ff
paper_authors.name: Nan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:23:24.483Z
paper_authors._id: 653a6b979d53a3f8513ca018
paper_authors.avatarUrl: /avatars/5aac9ce4f82629a6304c11273fe23f68.svg
paper_authors.fullname: Xu
paper_authors.type: user
paper_authors.user: Nanxu024
paper_authors._id: 667be60309cb8b5498ec9100
paper_authors.name: Lei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-19T07:08:30.331Z
paper_authors._id: 64c38871f9cd765462fa1a17
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c38871f9cd765462fa1a17/yuIlVcqeDlQVKsUF8uEl3.jpeg
paper_authors.fullname: Lei Zhang
paper_authors.type: user
paper_authors.user: Lemoncoke
paper_authors._id: 667be60309cb8b5498ec9101
paper_authors.name: Run Luo
paper_authors._id: 667be60309cb8b5498ec9102
paper_authors.name: Yunshui Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:23:55.566Z
paper_authors._id: 62e670d33651180f7d334ef3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661829952739-62e670d33651180f7d334ef3.jpeg
paper_authors.fullname: LiYunshui
paper_authors.type: user
paper_authors.user: Wa2erGo
paper_authors._id: 667be60309cb8b5498ec9103
paper_authors.name: Min Yang
paper_authors._id: 667be60309cb8b5498ec9104
paper_authors.name: Fei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:24:22.098Z
paper_authors._id: 635b8b6a37c6a2c12e2cce00
paper_authors.avatarUrl: /avatars/229fb72180529141515d1df797b33709.svg
paper_authors.fullname: Fei Huang
paper_authors.type: user
paper_authors.user: hzhwcmhf
paper_authors._id: 667be60309cb8b5498ec9105
paper_authors.name: Yongbin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T10:48:00.933Z
paper_authors._id: 66641b2fd8e1e34bc621e688
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
paper_authors.fullname: Yongbin Li
paper_authors.type: user
paper_authors.user: Yongbin-Li
paper_title: Leave No Document Behind: Benchmarking Long-Context LLMs with Extended
    Multi-Doc QA
paper_summary: Long-context modeling capabilities have garnered widespread attention,
  leading to the emergence of Large Language Models (LLMs) with ultra-context
  windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually
  catching up. However, existing benchmarks employ irrelevant noise texts to
  artificially extend the length of test cases, diverging from the real-world
  scenarios of long-context applications. To bridge this gap, we propose a novel
  long-context benchmark, Loong, aligning with realistic scenarios through
  extended multi-document question answering (QA). Unlike typical document QA, in
  Loong's test cases, each document is relevant to the final answer, ignoring any
  document will lead to the failure of the answer. Furthermore, Loong introduces
  four types of tasks with a range of context lengths: Spotlight Locating,
  Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic
  and comprehensive evaluation of long-context understanding. Extensive
  experiments indicate that existing long-context language models still exhibit
  considerable potential for enhancement. Retrieval augmented generation (RAG)
  achieves poor performance, demonstrating that Loong can reliably assess the
  model's long-context modeling capabilities.
paper_ai_keywords: Large Language Models (LLMs) ultra-context windows long-context LLMs extended multi-document question answering (QA) Shapley Spotlight Locating Comparison Clustering Chain of Reasoning Retrieval augmented generation (RAG)