title: BABILong: Testing the Limits of LLMs with Long Context
    Reasoning-in-a-Haystack
summary: In recent years, the input context sizes of large language models (LLMs) have
  increased dramatically. However, existing evaluation methods have not kept
  pace, failing to comprehensively assess the efficiency of models in handling
  long contexts. To bridge this gap, we introduce the BABILong benchmark,
  designed to test language models' ability to reason across facts distributed in
  extremely long documents. BABILong includes a diverse set of 20 reasoning
  tasks, including fact chaining, simple induction, deduction, counting, and
  handling lists/sets. These tasks are challenging on their own, and even more
  demanding when the required facts are scattered across long natural text. Our
  evaluations show that popular LLMs effectively utilize only 10-20\% of the
  context and their performance declines sharply with increased reasoning
  complexity. Among alternatives to in-context reasoning, Retrieval-Augmented
  Generation methods achieve a modest 60\% accuracy on single-fact question
  answering, independent of context length. Among context extension methods, the
  highest performance is demonstrated by recurrent memory transformers, enabling
  the processing of lengths up to 11 million tokens. The BABILong benchmark is
  extendable to any length to support the evaluation of new upcoming models with
  increased capabilities, and we provide splits up to 1 million token lengths.
paper_authors._id: 666ffb662b9e452739047aa9
paper_authors.name: Yuri Kuratov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T13:43:02.813Z
paper_authors._id: 618b9540682ec1c38327e586
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/618b9540682ec1c38327e586/v_ZBkfh8O9Zh6C2YQpuBX.jpeg
paper_authors.fullname: Yury Kuratov
paper_authors.type: user
paper_authors.user: yurakuratov
paper_authors._id: 666ffb662b9e452739047aaa
paper_authors.name: Aydar Bulatov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:13:46.699Z
paper_authors._id: 64c8b321cb2f1bf0e7c0f54b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c8b321cb2f1bf0e7c0f54b/JflXxMVnG9I0IB5YNyhXF.jpeg
paper_authors.fullname: Aydar Bulatov
paper_authors.type: user
paper_authors.user: booydar
paper_authors._id: 666ffb662b9e452739047aab
paper_authors.name: Petr Anokhin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:13:53.063Z
paper_authors._id: 64f734b08e7fa6529fb693f1
paper_authors.avatarUrl: /avatars/a50c7e1d22174f06fbaf99e8caca7c37.svg
paper_authors.fullname: Petr Anokhin
paper_authors.type: user
paper_authors.user: petranokhin
paper_authors._id: 666ffb662b9e452739047aac
paper_authors.name: Ivan Rodkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:13:59.696Z
paper_authors._id: 63c1ac8cc58fcfeac186bda2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c1ac8cc58fcfeac186bda2/MajMhLgsvCBN81tmO9PZc.jpeg
paper_authors.fullname: Ivan Rodkin
paper_authors.type: user
paper_authors.user: irodkin
paper_authors._id: 666ffb662b9e452739047aad
paper_authors.name: Dmitry Sorokin
paper_authors._id: 666ffb662b9e452739047aae
paper_authors.name: Artyom Sorokin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:14:09.088Z
paper_authors._id: 64beb75986e7fb5b8a8fa731
paper_authors.avatarUrl: /avatars/7053c64142300fa446ba5870eb28c3ee.svg
paper_authors.fullname: Artyom Sorokin
paper_authors.type: user
paper_authors.user: griver
paper_authors._id: 666ffb662b9e452739047aaf
paper_authors.name: Mikhail Burtsev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:14:16.679Z
paper_authors._id: 639c6e978a34ed9a404c6a7b
paper_authors.avatarUrl: /avatars/c98ca8c9f9ed8509c2f1bb6aa994fd57.svg
paper_authors.fullname: MIKHAIL BURTSEV
paper_authors.type: user
paper_authors.user: mbur
paper_title: BABILong: Testing the Limits of LLMs with Long Context
    Reasoning-in-a-Haystack
paper_summary: In recent years, the input context sizes of large language models (LLMs) have
  increased dramatically. However, existing evaluation methods have not kept
  pace, failing to comprehensively assess the efficiency of models in handling
  long contexts. To bridge this gap, we introduce the BABILong benchmark,
  designed to test language models' ability to reason across facts distributed in
  extremely long documents. BABILong includes a diverse set of 20 reasoning
  tasks, including fact chaining, simple induction, deduction, counting, and
  handling lists/sets. These tasks are challenging on their own, and even more
  demanding when the required facts are scattered across long natural text. Our
  evaluations show that popular LLMs effectively utilize only 10-20\% of the
  context and their performance declines sharply with increased reasoning
  complexity. Among alternatives to in-context reasoning, Retrieval-Augmented
  Generation methods achieve a modest 60\% accuracy on single-fact question
  answering, independent of context length. Among context extension methods, the
  highest performance is demonstrated by recurrent memory transformers, enabling
  the processing of lengths up to 11 million tokens. The BABILong benchmark is
  extendable to any length to support the evaluation of new upcoming models with
  increased capabilities, and we provide splits up to 1 million token lengths.
paper_ai_keywords: Retrieval-Augmented Generation recurrent memory transformers BABILong benchmark reasoning tasks fact chaining simple induction deduction counting handling lists/sets in-context reasoning context extension perplexity natural text lengthy tokens