Leave No Document Behind: Benchmarking Long-Context LLMs with Extended
  Multi-Doc QA
Long-context modeling capabilities have garnered widespread attention,
leading to the emergence of Large Language Models (LLMs) with ultra-context
windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually
catching up. However, existing benchmarks employ irrelevant noise texts to
artificially extend the length of test cases, diverging from the real-world
scenarios of long-context applications. To bridge this gap, we propose a novel
long-context benchmark, Loong, aligning with realistic scenarios through
extended multi-document question answering (QA). Unlike typical document QA, in
Loong's test cases, each document is relevant to the final answer, ignoring any
document will lead to the failure of the answer. Furthermore, Loong introduces
four types of tasks with a range of context lengths: Spotlight Locating,
Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic
and comprehensive evaluation of long-context understanding. Extensive
experiments indicate that existing long-context language models still exhibit
considerable potential for enhancement. Retrieval augmented generation (RAG)
achieves poor performance, demonstrating that Loong can reliably assess the
model's long-context modeling capabilities.
667be60309cb8b5498ec90f8
Minzheng Wang
claimed_verified
2024-06-26T10:18:23.390Z
64bcc373ef8c0e42bf16acc5
/avatars/873308203d28115ae1a9e4d0e26508f4.svg
mz.w
user
iiiiwis
667be60309cb8b5498ec90f9
Longze Chen
admin_assigned
2024-06-26T13:22:16.572Z
64c7b4d1c547ed5243c07b6c
https://cdn-avatars.huggingface.co/v1/production/uploads/64c7b4d1c547ed5243c07b6c/h96CLBj6dcm01soK2UJzr.jpeg
Longze Chen
user
lzchen2001
667be60309cb8b5498ec90fa
Cheng Fu
admin_assigned
2024-06-26T13:22:26.160Z
665813e398dd5097821b7df0
/avatars/baddbb04c38a6143224031f822e51f56.svg
cheng-fu
user
cfcloud
667be60309cb8b5498ec90fb
Shengyi Liao
admin_assigned
2024-06-26T13:22:37.397Z
64a69c1a9704e9888c1da827
/avatars/096aefab0ca99af76097cdfc5e752402.svg
LiaoShengyi
user
LsyLsyLsyyy
667be60309cb8b5498ec90fc
Xinghua Zhang
admin_assigned
2024-06-26T13:22:48.990Z
656426e7ec7e2398990a2d34
/avatars/e84aed6b55b0554ad9581ae3c138a16a.svg
AIRobotZ
user
AIRobotZ
667be60309cb8b5498ec90fd
Bingli Wu
admin_assigned
2024-06-26T13:22:59.506Z
64b75f7f00bac1088ceafc06
/avatars/8ce3ce9901d76ec96574769bdb11b8be.svg
wubingli
user
wubingli
667be60309cb8b5498ec90fe
Haiyang Yu
claimed_verified
2024-09-30T08:08:21.206Z
64a4ceda9a90f701134189b7
/avatars/859a189c5d2ae2fcb9aa2d79104fbfe7.svg
Haiyang Yu
user
yhycai
667be60309cb8b5498ec90ff
Nan Xu
admin_assigned
2024-06-26T13:23:24.483Z
653a6b979d53a3f8513ca018
/avatars/5aac9ce4f82629a6304c11273fe23f68.svg
Xu
user
Nanxu024
667be60309cb8b5498ec9100
Lei Zhang
claimed_verified
2024-09-19T07:08:30.331Z
64c38871f9cd765462fa1a17
https://cdn-avatars.huggingface.co/v1/production/uploads/64c38871f9cd765462fa1a17/yuIlVcqeDlQVKsUF8uEl3.jpeg
Lei Zhang
user
Lemoncoke
667be60309cb8b5498ec9101
Run Luo
667be60309cb8b5498ec9102
Yunshui Li
admin_assigned
2024-06-26T13:23:55.566Z
62e670d33651180f7d334ef3
https://cdn-avatars.huggingface.co/v1/production/uploads/1661829952739-62e670d33651180f7d334ef3.jpeg
Li Yunshui
user
Wa2erGo
667be60309cb8b5498ec9103
Min Yang
667be60309cb8b5498ec9104
Fei Huang
admin_assigned
2024-06-26T13:24:22.098Z
635b8b6a37c6a2c12e2cce00
/avatars/229fb72180529141515d1df797b33709.svg
Fei Huang
user
hzhwcmhf
667be60309cb8b5498ec9105
Yongbin Li
claimed_verified
2024-10-14T10:48:00.933Z
66641b2fd8e1e34bc621e688
https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
Yongbin Li
user
Yongbin-Li
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended
  Multi-Doc QA
Long-context modeling capabilities have garnered widespread attention,
leading to the emergence of Large Language Models (LLMs) with ultra-context
windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually
catching up. However, existing benchmarks employ irrelevant noise texts to
artificially extend the length of test cases, diverging from the real-world
scenarios of long-context applications. To bridge this gap, we propose a novel
long-context benchmark, Loong, aligning with realistic scenarios through
extended multi-document question answering (QA). Unlike typical document QA, in
Loong's test cases, each document is relevant to the final answer, ignoring any
document will lead to the failure of the answer. Furthermore, Loong introduces
four types of tasks with a range of context lengths: Spotlight Locating,
Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic
and comprehensive evaluation of long-context understanding. Extensive
experiments indicate that existing long-context language models still exhibit
considerable potential for enhancement. Retrieval augmented generation (RAG)
achieves poor performance, demonstrating that Loong can reliably assess the
model's long-context modeling capabilities.
A new long-context benchmark, Loong, evaluates large language models using extended multi-document question answering tasks that mimic real-world scenarios.
Large Language Models ultra-context windows long-context benchmarks noise texts extended multi-document question answering Spotlight Locating Comparison Clustering Chain of Reasoning retrieval augmented generation long-context modeling capabilities