title: LongBench v2: Towards Deeper Understanding and Reasoning on Realistic
    Long-context Multitasks
summary: This paper introduces LongBench v2, a benchmark designed to assess the
  ability of LLMs to handle long-context problems requiring deep understanding
  and reasoning across real-world multitasks. LongBench v2 consists of 503
  challenging multiple-choice questions, with contexts ranging from 8k to 2M
  words, across six major task categories: single-document QA, multi-document QA,
  long in-context learning, long-dialogue history understanding, code repository
  understanding, and long structured data understanding. To ensure the breadth
  and the practicality, we collect data from nearly 100 highly educated
  individuals with diverse professional backgrounds. We employ both automated and
  manual review processes to maintain high quality and difficulty, resulting in
  human experts achieving only 53.7% accuracy under a 15-minute time constraint.
  Our evaluation reveals that the best-performing model, when directly answers
  the questions, achieves only 50.1% accuracy. In contrast, the o1-preview model,
  which includes longer reasoning, achieves 57.7%, surpassing the human baseline
  by 4%. These results highlight the importance of enhanced reasoning ability and
  scaling inference-time compute to tackle the long-context challenges in
  LongBench v2. The project is available at https://longbench2.github.io.
paper_authors._id: 6764d4d46be739a319253381
paper_authors.name: Yushi Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:33:05.814Z
paper_authors._id: 64ed568ccf6118a9379a61b8
paper_authors.avatarUrl: /avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
paper_authors.fullname: Yushi Bai
paper_authors.type: user
paper_authors.user: bys0318
paper_authors._id: 6764d4d46be739a319253382
paper_authors.name: Shangqing Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:33:03.700Z
paper_authors._id: 648c48d8c0ddeee6df5b6d22
paper_authors.avatarUrl: /avatars/8706b0b16dfc332b96c91d3ced31bd0b.svg
paper_authors.fullname: Shangqing Tu
paper_authors.type: user
paper_authors.user: tsq2000
paper_authors._id: 6764d4d46be739a319253383
paper_authors.name: Jiajie Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:20:03.123Z
paper_authors._id: 66cdd285c51a915bd5f2d017
paper_authors.avatarUrl: /avatars/14e5794307e4672b1b51d26b31227e0f.svg
paper_authors.fullname: Jiajie Zhang
paper_authors.type: user
paper_authors.user: NeoZ123
paper_authors._id: 6764d4d46be739a319253384
paper_authors.name: Hao Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:19:34.830Z
paper_authors._id: 660ec5a2509153ca49775a7c
paper_authors.avatarUrl: /avatars/97570fc245cc8ec7628da9c13bd35b71.svg
paper_authors.fullname: Hao Peng
paper_authors.type: user
paper_authors.user: haopeng01
paper_authors._id: 6764d4d46be739a319253385
paper_authors.name: Xiaozhi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:19:21.316Z
paper_authors._id: 648c48e7a47a0850f9458b67
paper_authors.avatarUrl: /avatars/47d71d80f9901313feb0199c37296389.svg
paper_authors.fullname: Xiaozhi Wang
paper_authors.type: user
paper_authors.user: wangxz098
paper_authors._id: 6764d4d46be739a319253386
paper_authors.name: Xin Lv
paper_authors._id: 6764d4d46be739a319253387
paper_authors.name: Shulin Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:19:04.990Z
paper_authors._id: 62c924a6334a6ee11c2e8dfa
paper_authors.avatarUrl: /avatars/3dbc37af162b94d68cb83665ac4528c3.svg
paper_authors.fullname: ShulinCao
paper_authors.type: user
paper_authors.user: caoshulin
paper_authors._id: 6764d4d46be739a319253388
paper_authors.name: Jiazheng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:18:59.503Z
paper_authors._id: 62d7b131f6e8ba66107af761
paper_authors.avatarUrl: /avatars/f1c5df47aef69c824fd166722df8f670.svg
paper_authors.fullname: Jiazheng Xu
paper_authors.type: user
paper_authors.user: xujz0703
paper_authors._id: 6764d4d46be739a319253389
paper_authors.name: Lei Hou
paper_authors._id: 6764d4d46be739a31925338a
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:18:30.529Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_authors._id: 6764d4d46be739a31925338b
paper_authors.name: Jie Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:18:23.208Z
paper_authors._id: 640dff05474aa6f89556677e
paper_authors.avatarUrl: /avatars/1b4591c7322d649c797b3125148f1915.svg
paper_authors.fullname: Jie Tang
paper_authors.type: user
paper_authors.user: jerytang
paper_authors._id: 6764d4d46be739a31925338c
paper_authors.name: Juanzi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:18:13.826Z
paper_authors._id: 65df8cbc2705d9672f55d1aa
paper_authors.avatarUrl: /avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
paper_authors.fullname: Juanzi Li
paper_authors.type: user
paper_authors.user: juanli
paper_title: LongBench v2: Towards Deeper Understanding and Reasoning on Realistic
    Long-context Multitasks
paper_summary: This paper introduces LongBench v2, a benchmark designed to assess the
  ability of LLMs to handle long-context problems requiring deep understanding
  and reasoning across real-world multitasks. LongBench v2 consists of 503
  challenging multiple-choice questions, with contexts ranging from 8k to 2M
  words, across six major task categories: single-document QA, multi-document QA,
  long in-context learning, long-dialogue history understanding, code repository
  understanding, and long structured data understanding. To ensure the breadth
  and the practicality, we collect data from nearly 100 highly educated
  individuals with diverse professional backgrounds. We employ both automated and
  manual review processes to maintain high quality and difficulty, resulting in
  human experts achieving only 53.7% accuracy under a 15-minute time constraint.
  Our evaluation reveals that the best-performing model, when directly answers
  the questions, achieves only 50.1% accuracy. In contrast, the o1-preview model,
  which includes longer reasoning, achieves 57.7%, surpassing the human baseline
  by 4%. These results highlight the importance of enhanced reasoning ability and
  scaling inference-time compute to tackle the long-context challenges in
  LongBench v2. The project is available at https://longbench2.github.io.
paper_ai_summary: A benchmark evaluates large language models' performance on long-context problems across various tasks, showing that enhanced reasoning and increased computational resources improve accuracy beyond human performance.
paper_ai_keywords: LLMs multiple-choice questions single-document QA multi-document QA long in-context learning long-dialogue history understanding code repository understanding long structured data understanding human experts automated review manual review reasoning ability inference-time compute o1-preview model