title: Associative Recurrent Memory Transformer
summary: This paper addresses the challenge of creating a neural architecture for very
  long sequences that requires constant time for processing new information at
  each time step. Our approach, Associative Recurrent Memory Transformer (ARMT),
  is based on transformer self-attention for local context and segment-level
  recurrence for storage of task specific information distributed over a long
  context. We demonstrate that ARMT outperfors existing alternatives in
  associative retrieval tasks and sets a new performance record in the recent
  BABILong multi-task long-context benchmark by answering single-fact questions
  over 50 million tokens with an accuracy of 79.9%. The source code for training
  and evaluation is available on github.
paper_authors._id: 668cee02f8e6657dff5baeb7
paper_authors.name: Ivan Rodkin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T21:09:12.111Z
paper_authors._id: 63c1ac8cc58fcfeac186bda2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c1ac8cc58fcfeac186bda2/MajMhLgsvCBN81tmO9PZc.jpeg
paper_authors.fullname: Ivan Rodkin
paper_authors.type: user
paper_authors.user: irodkin
paper_authors._id: 668cee02f8e6657dff5baeb8
paper_authors.name: Yuri Kuratov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T08:17:45.419Z
paper_authors._id: 618b9540682ec1c38327e586
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/618b9540682ec1c38327e586/v_ZBkfh8O9Zh6C2YQpuBX.jpeg
paper_authors.fullname: Yury Kuratov
paper_authors.type: user
paper_authors.user: yurakuratov
paper_authors._id: 668cee02f8e6657dff5baeb9
paper_authors.name: Aydar Bulatov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T14:43:30.781Z
paper_authors._id: 64c8b321cb2f1bf0e7c0f54b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c8b321cb2f1bf0e7c0f54b/JflXxMVnG9I0IB5YNyhXF.jpeg
paper_authors.fullname: Aydar Bulatov
paper_authors.type: user
paper_authors.user: booydar
paper_authors._id: 668cee02f8e6657dff5baeba
paper_authors.name: Mikhail Burtsev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:08:18.446Z
paper_authors._id: 639c6e978a34ed9a404c6a7b
paper_authors.avatarUrl: /avatars/c98ca8c9f9ed8509c2f1bb6aa994fd57.svg
paper_authors.fullname: MIKHAIL BURTSEV
paper_authors.type: user
paper_authors.user: mbur
paper_title: Associative Recurrent Memory Transformer
paper_summary: This paper addresses the challenge of creating a neural architecture for very
  long sequences that requires constant time for processing new information at
  each time step. Our approach, Associative Recurrent Memory Transformer (ARMT),
  is based on transformer self-attention for local context and segment-level
  recurrence for storage of task specific information distributed over a long
  context. We demonstrate that ARMT outperfors existing alternatives in
  associative retrieval tasks and sets a new performance record in the recent
  BABILong multi-task long-context benchmark by answering single-fact questions
  over 50 million tokens with an accuracy of 79.9%. The source code for training
  and evaluation is available on github.
paper_ai_keywords: Associative Recurrent Memory Transformer (ARMT) transformer self-attention associative retrieval tasks BABILong multi-task long-context benchmark single-fact questions