title: LongIns: A Challenging Long-context Instruction-based Exam for LLMs
summary: The long-context capabilities of large language models (LLMs) have been a hot
  topic in recent years. To evaluate the performance of LLMs in different
  scenarios, various assessment benchmarks have emerged. However, as most of
  these benchmarks focus on identifying key information to answer questions,
  which mainly requires the retrieval ability of LLMs, these benchmarks can
  partially represent the reasoning performance of LLMs from large amounts of
  information. Meanwhile, although LLMs often claim to have context windows of
  32k, 128k, 200k, or even longer, these benchmarks fail to reveal the actual
  supported length of these LLMs. To address these issues, we propose the LongIns
  benchmark dataset, a challenging long-context instruction-based exam for LLMs,
  which is built based on the existing instruction datasets. Specifically, in our
  LongIns, we introduce three evaluation settings: Global Instruction & Single
  Task (GIST), Local Instruction & Single Task (LIST), and Local Instruction &
  Multiple Tasks (LIMT). Based on LongIns, we perform comprehensive evaluations
  on existing LLMs and have the following important findings: (1). The
  top-performing GPT-4 with 128k context length performs poorly on the evaluation
  context window of 16k in our LongIns. (2). For the multi-hop reasoning ability
  of many existing LLMs, significant efforts are still needed under short context
  windows (less than 4k).
paper_authors._id: 667b958491bd7b3e1e04bdc5
paper_authors.name: Shawn Gavin
paper_authors._id: 667b958491bd7b3e1e04bdc6
paper_authors.name: Tuney Zheng
paper_authors._id: 667b958491bd7b3e1e04bdc7
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T09:55:28.515Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 667b958491bd7b3e1e04bdc8
paper_authors.name: Quehry Que
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T09:56:11.687Z
paper_authors._id: 632176664204950905b64e58
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632176664204950905b64e58/IzkGIaAWy9285Q8uBncZ6.jpeg
paper_authors.fullname: quehry
paper_authors.type: user
paper_authors.user: quehry
paper_authors._id: 667b958491bd7b3e1e04bdc9
paper_authors.name: Noah Wang
paper_authors._id: 667b958491bd7b3e1e04bdca
paper_authors.name: Jian Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:15:14.541Z
paper_authors._id: 64ccb9bfead94891d12aef42
paper_authors.avatarUrl: /avatars/c54809d43d93d3f0766bd2555cecc4e3.svg
paper_authors.fullname: Yang Jian
paper_authors.type: user
paper_authors.user: CSJianYang
paper_authors._id: 667b958491bd7b3e1e04bdcb
paper_authors.name: Chenchen Zhang
paper_authors._id: 667b958491bd7b3e1e04bdcc
paper_authors.name: Wenhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:30:14.083Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 667b958491bd7b3e1e04bdcd
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T09:57:00.522Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_authors._id: 667b958491bd7b3e1e04bdce
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:03:11.978Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_title: LongIns: A Challenging Long-context Instruction-based Exam for LLMs
paper_summary: The long-context capabilities of large language models (LLMs) have been a hot
  topic in recent years. To evaluate the performance of LLMs in different
  scenarios, various assessment benchmarks have emerged. However, as most of
  these benchmarks focus on identifying key information to answer questions,
  which mainly requires the retrieval ability of LLMs, these benchmarks can
  partially represent the reasoning performance of LLMs from large amounts of
  information. Meanwhile, although LLMs often claim to have context windows of
  32k, 128k, 200k, or even longer, these benchmarks fail to reveal the actual
  supported length of these LLMs. To address these issues, we propose the LongIns
  benchmark dataset, a challenging long-context instruction-based exam for LLMs,
  which is built based on the existing instruction datasets. Specifically, in our
  LongIns, we introduce three evaluation settings: Global Instruction & Single
  Task (GIST), Local Instruction & Single Task (LIST), and Local Instruction &
  Multiple Tasks (LIMT). Based on LongIns, we perform comprehensive evaluations
  on existing LLMs and have the following important findings: (1). The
  top-performing GPT-4 with 128k context length performs poorly on the evaluation
  context window of 16k in our LongIns. (2). For the multi-hop reasoning ability
  of many existing LLMs, significant efforts are still needed under short context
  windows (less than 4k).
paper_ai_keywords: long-context capabilities large language models (LLMs) assessment benchmarks retrieval ability reasoning performance context windows LongIns benchmark dataset Global Instruction & Single Task (GIST) Local Instruction & Single Task (LIST) Local Instruction & Multiple Tasks (LIMT) multi-hop reasoning ability