title: In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs
    Miss
summary: This paper addresses the challenge of processing long documents using
  generative transformer models. To evaluate different approaches, we introduce
  BABILong, a new benchmark designed to assess model capabilities in extracting
  and processing distributed facts within extensive texts. Our evaluation, which
  includes benchmarks for GPT-4 and RAG, reveals that common methods are
  effective only for sequences up to 10^4 elements. In contrast, fine-tuning
  GPT-2 with recurrent memory augmentations enables it to handle tasks involving
  up to 10^7 elements. This achievement marks a substantial leap, as it is by
  far the longest input processed by any open neural network model to date,
  demonstrating a significant improvement in the processing capabilities for long
  sequences.
paper_authors._id: 65d2d4b76f1a8bf6a1a09c2e
paper_authors.name: Yuri Kuratov
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-19T14:28:27.723Z
paper_authors._id: 618b9540682ec1c38327e586
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/618b9540682ec1c38327e586/v_ZBkfh8O9Zh6C2YQpuBX.jpeg
paper_authors.fullname: Yury Kuratov
paper_authors.type: user
paper_authors.user: yurakuratov
paper_authors._id: 65d2d4b76f1a8bf6a1a09c2f
paper_authors.name: Aydar Bulatov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-19T14:56:06.851Z
paper_authors._id: 64c8b321cb2f1bf0e7c0f54b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c8b321cb2f1bf0e7c0f54b/JflXxMVnG9I0IB5YNyhXF.jpeg
paper_authors.fullname: Aydar Bulatov
paper_authors.type: user
paper_authors.user: booydar
paper_authors._id: 65d2d4b76f1a8bf6a1a09c30
paper_authors.name: Petr Anokhin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-19T14:53:14.986Z
paper_authors._id: 64f734b08e7fa6529fb693f1
paper_authors.avatarUrl: /avatars/a50c7e1d22174f06fbaf99e8caca7c37.svg
paper_authors.fullname: Petr Anokhin
paper_authors.type: user
paper_authors.user: petranokhin
paper_authors._id: 65d2d4b76f1a8bf6a1a09c31
paper_authors.name: Dmitry Sorokin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-29T17:18:02.324Z
paper_authors._id: 635bdcfd86f52b24c1e70e31
paper_authors.avatarUrl: /avatars/8ade00f3e4f25b1cb52ddc0740087fad.svg
paper_authors.fullname: Dmitry Sorokin
paper_authors.type: user
paper_authors.user: dsorokin
paper_authors._id: 65d2d4b76f1a8bf6a1a09c32
paper_authors.name: Artyom Sorokin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:01:45.476Z
paper_authors._id: 64beb75986e7fb5b8a8fa731
paper_authors.avatarUrl: /avatars/7053c64142300fa446ba5870eb28c3ee.svg
paper_authors.fullname: Artyom Sorokin
paper_authors.type: user
paper_authors.user: griver
paper_authors._id: 65d2d4b76f1a8bf6a1a09c33
paper_authors.name: Mikhail Burtsev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:01:51.577Z
paper_authors._id: 639c6e978a34ed9a404c6a7b
paper_authors.avatarUrl: /avatars/c98ca8c9f9ed8509c2f1bb6aa994fd57.svg
paper_authors.fullname: MIKHAIL BURTSEV
paper_authors.type: user
paper_authors.user: mbur
paper_title: In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs
    Miss
paper_summary: This paper addresses the challenge of processing long documents using
  generative transformer models. To evaluate different approaches, we introduce
  BABILong, a new benchmark designed to assess model capabilities in extracting
  and processing distributed facts within extensive texts. Our evaluation, which
  includes benchmarks for GPT-4 and RAG, reveals that common methods are
  effective only for sequences up to 10^4 elements. In contrast, fine-tuning
  GPT-2 with recurrent memory augmentations enables it to handle tasks involving
  up to 10^7 elements. This achievement marks a substantial leap, as it is by
  far the longest input processed by any open neural network model to date,
  demonstrating a significant improvement in the processing capabilities for long
  sequences.
paper_ai_keywords: generative transformer models BABILong distributed facts sequences fine-tuning recurrent memory augmentations open neural network model