title: Needle Threading: Can LLMs Follow Threads through Near-Million-Scale
    Haystacks?
summary: As the context limits of Large Language Models (LLMs) increase, the range of
  possible applications and downstream functions broadens. In many real-world
  tasks, decisions depend on details scattered across collections of often
  disparate documents containing mostly irrelevant information. Long-context LLMs
  appear well-suited to this form of complex information retrieval and reasoning,
  which has traditionally proven costly and time-consuming. However, although the
  development of longer context models has seen rapid gains in recent years, our
  understanding of how effectively LLMs use their context has not kept pace. To
  address this, we conduct a set of retrieval experiments designed to evaluate
  the capabilities of 17 leading LLMs, such as their ability to follow threads of
  information through the context window. Strikingly, we find that many models
  are remarkably threadsafe: capable of simultaneously following multiple threads
  without significant loss in performance. Still, for many models, we find the
  effective context limit is significantly shorter than the supported context
  length, with accuracy decreasing as the context window grows. Our study also
  highlights the important point that token counts from different tokenizers
  should not be directly compared -- they often correspond to substantially
  different numbers of written characters. We release our code and long-context
  experimental data.
paper_authors._id: 672deabffbdaf22d9db944c0
paper_authors.name: Jonathan Roberts
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:29:08.971Z
paper_authors._id: 632456d21ed511c0c5231afd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671535517748-632456d21ed511c0c5231afd.jpeg
paper_authors.fullname: Jonathan Roberts
paper_authors.type: user
paper_authors.user: jonathan-roberts1
paper_authors._id: 672deabffbdaf22d9db944c1
paper_authors.name: Kai Han
paper_authors._id: 672deabffbdaf22d9db944c2
paper_authors.name: Samuel Albanie
paper_title: Needle Threading: Can LLMs Follow Threads through Near-Million-Scale
    Haystacks?
paper_summary: As the context limits of Large Language Models (LLMs) increase, the range of
  possible applications and downstream functions broadens. In many real-world
  tasks, decisions depend on details scattered across collections of often
  disparate documents containing mostly irrelevant information. Long-context LLMs
  appear well-suited to this form of complex information retrieval and reasoning,
  which has traditionally proven costly and time-consuming. However, although the
  development of longer context models has seen rapid gains in recent years, our
  understanding of how effectively LLMs use their context has not kept pace. To
  address this, we conduct a set of retrieval experiments designed to evaluate
  the capabilities of 17 leading LLMs, such as their ability to follow threads of
  information through the context window. Strikingly, we find that many models
  are remarkably threadsafe: capable of simultaneously following multiple threads
  without significant loss in performance. Still, for many models, we find the
  effective context limit is significantly shorter than the supported context
  length, with accuracy decreasing as the context window grows. Our study also
  highlights the important point that token counts from different tokenizers
  should not be directly compared -- they often correspond to substantially
  different numbers of written characters. We release our code and long-context
  experimental data.
paper_ai_summary: Research evaluates the information retrieval and reasoning capabilities of 17 long-context LLMs, revealing that many can handle multiple threads of information efficiently within a limited effective context window.
paper_ai_keywords: LLMs long-context models context window information retrieval reasoning threadsafe tokenizers