title: LM2: Large Memory Models
summary: This paper introduces the Large Memory Model (LM2), a decoder-only
  Transformer architecture enhanced with an auxiliary memory module that aims to
  address the limitations of standard Transformers in multi-step reasoning,
  relational argumentation, and synthesizing information distributed over long
  contexts. The proposed LM2 incorporates a memory module that acts as a
  contextual representation repository, interacting with input tokens via cross
  attention and updating through gating mechanisms. To preserve the Transformers
  general-purpose capabilities, LM2 maintains the original information flow while
  integrating a complementary memory pathway. Experimental results on the
  BABILong benchmark demonstrate that the LM2model outperforms both the
  memory-augmented RMT model by 37.1% and the baseline Llama-3.2 model by 86.3%
  on average across tasks. LM2 exhibits exceptional capabilities in multi-hop
  inference, numerical reasoning, and large-context question-answering. On the
  MMLU dataset, it achieves a 5.0% improvement over a pre-trained vanilla model,
  demonstrating that its memory module does not degrade performance on general
  tasks. Further, in our analysis, we explore the memory interpretability,
  effectiveness of memory modules, and test-time behavior. Our findings emphasize
  the importance of explicit memory in enhancing Transformer architectures.
paper_authors._id: 67aac01bd7b18841e7c266df
paper_authors.name: Jikun Kang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T15:26:38.660Z
paper_authors._id: 6489e10ca13f65198dc6e122
paper_authors.avatarUrl: /avatars/4aa9eab488157711b2f0298ddadee2f4.svg
paper_authors.fullname: Kang
paper_authors.type: user
paper_authors.user: JaxonK
paper_authors._id: 67aac01bd7b18841e7c266e0
paper_authors.name: Wenqi Wu
paper_authors._id: 67aac01bd7b18841e7c266e1
paper_authors.name: Filippos Christianos
paper_authors._id: 67aac01bd7b18841e7c266e2
paper_authors.name: Alex J. Chan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T15:26:36.110Z
paper_authors._id: 636c1e4415cd58e915bc45df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636c1e4415cd58e915bc45df/KnPgdPe0G5ngvXaCBua6R.jpeg
paper_authors.fullname: Alex J. Chan
paper_authors.type: user
paper_authors.user: XanderJC
paper_authors._id: 67aac01bd7b18841e7c266e3
paper_authors.name: Fraser Greenlee
paper_authors._id: 67aac01bd7b18841e7c266e4
paper_authors.name: George Thomas
paper_authors._id: 67aac01bd7b18841e7c266e5
paper_authors.name: Marvin Purtorab
paper_authors._id: 67aac01bd7b18841e7c266e6
paper_authors.name: Andy Toulis
paper_title: LM2: Large Memory Models
paper_summary: This paper introduces the Large Memory Model (LM2), a decoder-only
  Transformer architecture enhanced with an auxiliary memory module that aims to
  address the limitations of standard Transformers in multi-step reasoning,
  relational argumentation, and synthesizing information distributed over long
  contexts. The proposed LM2 incorporates a memory module that acts as a
  contextual representation repository, interacting with input tokens via cross
  attention and updating through gating mechanisms. To preserve the Transformers
  general-purpose capabilities, LM2 maintains the original information flow while
  integrating a complementary memory pathway. Experimental results on the
  BABILong benchmark demonstrate that the LM2model outperforms both the
  memory-augmented RMT model by 37.1% and the baseline Llama-3.2 model by 86.3%
  on average across tasks. LM2 exhibits exceptional capabilities in multi-hop
  inference, numerical reasoning, and large-context question-answering. On the
  MMLU dataset, it achieves a 5.0% improvement over a pre-trained vanilla model,
  demonstrating that its memory module does not degrade performance on general
  tasks. Further, in our analysis, we explore the memory interpretability,
  effectiveness of memory modules, and test-time behavior. Our findings emphasize
  the importance of explicit memory in enhancing Transformer architectures.
paper_ai_keywords: Transformer architecture auxiliary memory module multi-step reasoning relational argumentation cross attention gating mechanisms context flow complementary memory pathway multi-hop inference numerical reasoning interpretability memory modules test-time behavior