NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context
  Window?
In evaluating the long-context capabilities of large language models (LLMs),
identifying content relevant to a user's query from original long documents is
a crucial prerequisite for any LLM to answer questions based on long text. We
present NeedleBench, a framework consisting of a series of progressively more
challenging tasks for assessing bilingual long-context capabilities, spanning
multiple length intervals (4k, 8k, 32k, 128k, 200k, 1000k, and beyond) and
different depth ranges, allowing the strategic insertion of critical data
points in different text depth zones to rigorously test the retrieval and
reasoning capabilities of models in diverse contexts. We use the NeedleBench
framework to assess how well the leading open-source models can identify key
information relevant to the question and apply that information to reasoning in
bilingual long texts. Furthermore, we propose the Ancestral Trace Challenge
(ATC) to mimic the complexity of logical reasoning challenges that are likely
to be present in real-world long-context tasks, providing a simple method for
evaluating LLMs in dealing with complex long-context situations. Our results
suggest that current LLMs have significant room for improvement in practical
long-context applications, as they struggle with the complexity of logical
reasoning challenges that are likely to be present in real-world long-context
tasks. All codes and resources are available at OpenCompass:
https://github.com/open-compass/opencompass.
66973c07eb2a785068bb60ae
Mo Li
claimed_verified
2024-07-25T12:21:44.354Z
6576fe2b42ab083faea19841
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/c91ZKOR2E0gL8iIVkvEUa.jpeg
Mo Li
user
Mor-Li
66973c07eb2a785068bb60af
Songyang Zhang
admin_assigned
2024-07-23T13:48:52.460Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
66973c07eb2a785068bb60b0
Yunxin Liu
66973c07eb2a785068bb60b1
Kai Chen
NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context
  Window?
In evaluating the long-context capabilities of large language models (LLMs),
identifying content relevant to a user's query from original long documents is
a crucial prerequisite for any LLM to answer questions based on long text. We
present NeedleBench, a framework consisting of a series of progressively more
challenging tasks for assessing bilingual long-context capabilities, spanning
multiple length intervals (4k, 8k, 32k, 128k, 200k, 1000k, and beyond) and
different depth ranges, allowing the strategic insertion of critical data
points in different text depth zones to rigorously test the retrieval and
reasoning capabilities of models in diverse contexts. We use the NeedleBench
framework to assess how well the leading open-source models can identify key
information relevant to the question and apply that information to reasoning in
bilingual long texts. Furthermore, we propose the Ancestral Trace Challenge
(ATC) to mimic the complexity of logical reasoning challenges that are likely
to be present in real-world long-context tasks, providing a simple method for
evaluating LLMs in dealing with complex long-context situations. Our results
suggest that current LLMs have significant room for improvement in practical
long-context applications, as they struggle with the complexity of logical
reasoning challenges that are likely to be present in real-world long-context
tasks. All codes and resources are available at OpenCompass:
https://github.com/open-compass/opencompass.
NeedleBench and the Ancestral Trace Challenge evaluate the long-context retrieval and reasoning capabilities of large language models, highlighting their limitations in practical applications.
large language models LLMS NeedleBench bilingual long-context capabilities retrieval capabilities reasoning capabilities Ancestral Trace Challenge logical reasoning challenges