LM2: Large Memory Models
This paper introduces the Large Memory Model (LM2), a decoder-only
Transformer architecture enhanced with an auxiliary memory module that aims to
address the limitations of standard Transformers in multi-step reasoning,
relational argumentation, and synthesizing information distributed over long
contexts. The proposed LM2 incorporates a memory module that acts as a
contextual representation repository, interacting with input tokens via cross
attention and updating through gating mechanisms. To preserve the Transformers
general-purpose capabilities, LM2 maintains the original information flow while
integrating a complementary memory pathway. Experimental results on the
BABILong benchmark demonstrate that the LM2model outperforms both the
memory-augmented RMT model by 37.1% and the baseline Llama-3.2 model by 86.3%
on average across tasks. LM2 exhibits exceptional capabilities in multi-hop
inference, numerical reasoning, and large-context question-answering. On the
MMLU dataset, it achieves a 5.0% improvement over a pre-trained vanilla model,
demonstrating that its memory module does not degrade performance on general
tasks. Further, in our analysis, we explore the memory interpretability,
effectiveness of memory modules, and test-time behavior. Our findings emphasize
the importance of explicit memory in enhancing Transformer architectures.
67aac01bd7b18841e7c266df
Jikun Kang
claimed_verified
2025-02-11T15:26:38.660Z
6489e10ca13f65198dc6e122
/avatars/4aa9eab488157711b2f0298ddadee2f4.svg
Kang
user
JaxonK
67aac01bd7b18841e7c266e0
Wenqi Wu
67aac01bd7b18841e7c266e1
Filippos Christianos
67aac01bd7b18841e7c266e2
Alex J. Chan
claimed_verified
2025-02-11T15:26:36.110Z
636c1e4415cd58e915bc45df
https://cdn-avatars.huggingface.co/v1/production/uploads/636c1e4415cd58e915bc45df/KnPgdPe0G5ngvXaCBua6R.jpeg
Alex J. Chan
user
XanderJC
67aac01bd7b18841e7c266e3
Fraser Greenlee
67aac01bd7b18841e7c266e4
George Thomas
67aac01bd7b18841e7c266e5
Marvin Purtorab
67aac01bd7b18841e7c266e6
Andy Toulis
LM2: Large Memory Models
This paper introduces the Large Memory Model (LM2), a decoder-only
Transformer architecture enhanced with an auxiliary memory module that aims to
address the limitations of standard Transformers in multi-step reasoning,
relational argumentation, and synthesizing information distributed over long
contexts. The proposed LM2 incorporates a memory module that acts as a
contextual representation repository, interacting with input tokens via cross
attention and updating through gating mechanisms. To preserve the Transformers
general-purpose capabilities, LM2 maintains the original information flow while
integrating a complementary memory pathway. Experimental results on the
BABILong benchmark demonstrate that the LM2model outperforms both the
memory-augmented RMT model by 37.1% and the baseline Llama-3.2 model by 86.3%
on average across tasks. LM2 exhibits exceptional capabilities in multi-hop
inference, numerical reasoning, and large-context question-answering. On the
MMLU dataset, it achieves a 5.0% improvement over a pre-trained vanilla model,
demonstrating that its memory module does not degrade performance on general
tasks. Further, in our analysis, we explore the memory interpretability,
effectiveness of memory modules, and test-time behavior. Our findings emphasize
the importance of explicit memory in enhancing Transformer architectures.
The Large Memory Model (LM2), a decoder-only Transformer with an auxiliary memory module, improves multi-step reasoning, relational argumentation, and large-context processing while preserving Transformer capabilities.
Transformer architecture memory module cross attention gating mechanisms multi-hop inference numerical reasoning large-context question-answering MMLU dataset memory interpretability memory module effectiveness test-time behavior