LongIns: A Challenging Long-context Instruction-based Exam for LLMs
The long-context capabilities of large language models (LLMs) have been a hot
topic in recent years. To evaluate the performance of LLMs in different
scenarios, various assessment benchmarks have emerged. However, as most of
these benchmarks focus on identifying key information to answer questions,
which mainly requires the retrieval ability of LLMs, these benchmarks can
partially represent the reasoning performance of LLMs from large amounts of
information. Meanwhile, although LLMs often claim to have context windows of
32k, 128k, 200k, or even longer, these benchmarks fail to reveal the actual
supported length of these LLMs. To address these issues, we propose the LongIns
benchmark dataset, a challenging long-context instruction-based exam for LLMs,
which is built based on the existing instruction datasets. Specifically, in our
LongIns, we introduce three evaluation settings: Global Instruction & Single
Task (GIST), Local Instruction & Single Task (LIST), and Local Instruction &
Multiple Tasks (LIMT). Based on LongIns, we perform comprehensive evaluations
on existing LLMs and have the following important findings: (1). The
top-performing GPT-4 with 128k context length performs poorly on the evaluation
context window of 16k in our LongIns. (2). For the multi-hop reasoning ability
of many existing LLMs, significant efforts are still needed under short context
windows (less than 4k).
667b958491bd7b3e1e04bdc5
Shawn Gavin
667b958491bd7b3e1e04bdc6
Tuney Zheng
667b958491bd7b3e1e04bdc7
Jiaheng Liu
admin_assigned
2024-06-26T09:55:28.515Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
667b958491bd7b3e1e04bdc8
Quehry Que
admin_assigned
2024-06-26T09:56:11.687Z
632176664204950905b64e58
https://cdn-avatars.huggingface.co/v1/production/uploads/632176664204950905b64e58/IzkGIaAWy9285Q8uBncZ6.jpeg
quehry
user
quehry
667b958491bd7b3e1e04bdc9
Noah Wang
667b958491bd7b3e1e04bdca
Jian Yang
claimed_verified
2024-10-14T19:15:14.541Z
64ccb9bfead94891d12aef42
/avatars/c54809d43d93d3f0766bd2555cecc4e3.svg
Yang Jian
user
CSJianYang
667b958491bd7b3e1e04bdcb
Chenchen Zhang
667b958491bd7b3e1e04bdcc
Wenhao Huang
claimed_verified
2024-10-13T20:30:14.083Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
667b958491bd7b3e1e04bdcd
Wenhu Chen
admin_assigned
2024-06-26T09:57:00.522Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
667b958491bd7b3e1e04bdce
Ge Zhang
claimed_verified
2024-06-26T08:03:11.978Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
LongIns: A Challenging Long-context Instruction-based Exam for LLMs
The long-context capabilities of large language models (LLMs) have been a hot
topic in recent years. To evaluate the performance of LLMs in different
scenarios, various assessment benchmarks have emerged. However, as most of
these benchmarks focus on identifying key information to answer questions,
which mainly requires the retrieval ability of LLMs, these benchmarks can
partially represent the reasoning performance of LLMs from large amounts of
information. Meanwhile, although LLMs often claim to have context windows of
32k, 128k, 200k, or even longer, these benchmarks fail to reveal the actual
supported length of these LLMs. To address these issues, we propose the LongIns
benchmark dataset, a challenging long-context instruction-based exam for LLMs,
which is built based on the existing instruction datasets. Specifically, in our
LongIns, we introduce three evaluation settings: Global Instruction & Single
Task (GIST), Local Instruction & Single Task (LIST), and Local Instruction &
Multiple Tasks (LIMT). Based on LongIns, we perform comprehensive evaluations
on existing LLMs and have the following important findings: (1). The
top-performing GPT-4 with 128k context length performs poorly on the evaluation
context window of 16k in our LongIns. (2). For the multi-hop reasoning ability
of many existing LLMs, significant efforts are still needed under short context
windows (less than 4k).
The LongIns benchmark evaluates large language models' long-context reasoning abilities, revealing performance issues even in models with large context windows.
LLMs long-context capabilities assessment benchmarks retrieval ability reasoning performance context windows Global Instruction & Single Task Local Instruction & Single Task Local Instruction & Multiple Tasks multi-hop reasoning ability