title: NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context
    Window?
summary: In evaluating the long-context capabilities of large language models (LLMs),
  identifying content relevant to a user's query from original long documents is
  a crucial prerequisite for any LLM to answer questions based on long text. We
  present NeedleBench, a framework consisting of a series of progressively more
  challenging tasks for assessing bilingual long-context capabilities, spanning
  multiple length intervals (4k, 8k, 32k, 128k, 200k, 1000k, and beyond) and
  different depth ranges, allowing the strategic insertion of critical data
  points in different text depth zones to rigorously test the retrieval and
  reasoning capabilities of models in diverse contexts. We use the NeedleBench
  framework to assess how well the leading open-source models can identify key
  information relevant to the question and apply that information to reasoning in
  bilingual long texts. Furthermore, we propose the Ancestral Trace Challenge
  (ATC) to mimic the complexity of logical reasoning challenges that are likely
  to be present in real-world long-context tasks, providing a simple method for
  evaluating LLMs in dealing with complex long-context situations. Our results
  suggest that current LLMs have significant room for improvement in practical
  long-context applications, as they struggle with the complexity of logical
  reasoning challenges that are likely to be present in real-world long-context
  tasks. All codes and resources are available at OpenCompass:
  https://github.com/open-compass/opencompass.
paper_authors._id: 66973c07eb2a785068bb60ae
paper_authors.name: Mo Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-25T12:21:44.354Z
paper_authors._id: 6576fe2b42ab083faea19841
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/c91ZKOR2E0gL8iIVkvEUa.jpeg
paper_authors.fullname: Mo Li
paper_authors.type: user
paper_authors.user: Mor-Li
paper_authors._id: 66973c07eb2a785068bb60af
paper_authors.name: Songyang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T13:48:52.460Z
paper_authors._id: 630716d11801ecc7d2595021
paper_authors.avatarUrl: /avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
paper_authors.fullname: Songyang Zhang
paper_authors.type: user
paper_authors.user: zsytony
paper_authors._id: 66973c07eb2a785068bb60b0
paper_authors.name: Yunxin Liu
paper_authors._id: 66973c07eb2a785068bb60b1
paper_authors.name: Kai Chen
paper_title: NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context
    Window?
paper_summary: In evaluating the long-context capabilities of large language models (LLMs),
  identifying content relevant to a user's query from original long documents is
  a crucial prerequisite for any LLM to answer questions based on long text. We
  present NeedleBench, a framework consisting of a series of progressively more
  challenging tasks for assessing bilingual long-context capabilities, spanning
  multiple length intervals (4k, 8k, 32k, 128k, 200k, 1000k, and beyond) and
  different depth ranges, allowing the strategic insertion of critical data
  points in different text depth zones to rigorously test the retrieval and
  reasoning capabilities of models in diverse contexts. We use the NeedleBench
  framework to assess how well the leading open-source models can identify key
  information relevant to the question and apply that information to reasoning in
  bilingual long texts. Furthermore, we propose the Ancestral Trace Challenge
  (ATC) to mimic the complexity of logical reasoning challenges that are likely
  to be present in real-world long-context tasks, providing a simple method for
  evaluating LLMs in dealing with complex long-context situations. Our results
  suggest that current LLMs have significant room for improvement in practical
  long-context applications, as they struggle with the complexity of logical
  reasoning challenges that are likely to be present in real-world long-context
  tasks. All codes and resources are available at OpenCompass:
  https://github.com/open-compass/opencompass.
paper_ai_keywords: NeedleBench bilingual long-context capabilities length intervals text depth zones retrieval reasoning capabilities Ancestral Trace Challenge (ATC) logical reasoning challenges