In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs
  Miss
This paper addresses the challenge of processing long documents using
generative transformer models. To evaluate different approaches, we introduce
BABILong, a new benchmark designed to assess model capabilities in extracting
and processing distributed facts within extensive texts. Our evaluation, which
includes benchmarks for GPT-4 and RAG, reveals that common methods are
effective only for sequences up to 10^4 elements. In contrast, fine-tuning
GPT-2 with recurrent memory augmentations enables it to handle tasks involving
up to 10^7 elements. This achievement marks a substantial leap, as it is by
far the longest input processed by any open neural network model to date,
demonstrating a significant improvement in the processing capabilities for long
sequences.
65d2d4b76f1a8bf6a1a09c2e
Yuri Kuratov
extracted_confirmed
2024-02-19T14:28:27.723Z
618b9540682ec1c38327e586
https://cdn-avatars.huggingface.co/v1/production/uploads/618b9540682ec1c38327e586/v_ZBkfh8O9Zh6C2YQpuBX.jpeg
Yury Kuratov
user
yurakuratov
65d2d4b76f1a8bf6a1a09c2f
Aydar Bulatov
claimed_verified
2024-02-19T14:56:06.851Z
64c8b321cb2f1bf0e7c0f54b
https://cdn-avatars.huggingface.co/v1/production/uploads/64c8b321cb2f1bf0e7c0f54b/JflXxMVnG9I0IB5YNyhXF.jpeg
Aydar Bulatov
user
booydar
65d2d4b76f1a8bf6a1a09c30
Petr Anokhin
claimed_verified
2024-02-19T14:53:14.986Z
64f734b08e7fa6529fb693f1
/avatars/a50c7e1d22174f06fbaf99e8caca7c37.svg
Petr Anokhin
user
petranokhin
65d2d4b76f1a8bf6a1a09c31
Dmitry Sorokin
claimed_verified
2024-02-29T17:18:02.324Z
635bdcfd86f52b24c1e70e31
/avatars/8ade00f3e4f25b1cb52ddc0740087fad.svg
Dmitry Sorokin
user
dsorokin
65d2d4b76f1a8bf6a1a09c32
Artyom Sorokin
admin_assigned
2024-02-19T11:01:45.476Z
64beb75986e7fb5b8a8fa731
/avatars/7053c64142300fa446ba5870eb28c3ee.svg
Artyom Sorokin
user
griver
65d2d4b76f1a8bf6a1a09c33
Mikhail Burtsev
admin_assigned
2024-02-19T11:01:51.577Z
639c6e978a34ed9a404c6a7b
/avatars/c98ca8c9f9ed8509c2f1bb6aa994fd57.svg
MIKHAIL BURTSEV
user
mbur
In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs
  Miss
This paper addresses the challenge of processing long documents using
generative transformer models. To evaluate different approaches, we introduce
BABILong, a new benchmark designed to assess model capabilities in extracting
and processing distributed facts within extensive texts. Our evaluation, which
includes benchmarks for GPT-4 and RAG, reveals that common methods are
effective only for sequences up to 10^4 elements. In contrast, fine-tuning
GPT-2 with recurrent memory augmentations enables it to handle tasks involving
up to 10^7 elements. This achievement marks a substantial leap, as it is by
far the longest input processed by any open neural network model to date,
demonstrating a significant improvement in the processing capabilities for long
sequences.
Recursively augmented GPT-2 fine-tuning significantly extends the processing capability of generative models to handle long document sequences beyond $10^6$ elements.
generative transformer models BABILong benchmark GPT-4 RAG fine-tuning recurrent memory open neural network models