title: MoM: Linear Sequence Modeling with Mixture-of-Memories
summary: Linear sequence modeling methods, such as linear attention, state space
  modeling, and linear RNNs, offer significant efficiency improvements by
  reducing the complexity of training and inference. However, these methods
  typically compress the entire input sequence into a single fixed-size memory
  state, which leads to suboptimal performance on recall-intensive downstream
  tasks. Drawing inspiration from neuroscience, particularly the brain's ability
  to maintain robust long-term memory while mitigating "memory interference", we
  introduce a novel architecture called Mixture-of-Memories (MoM). MoM utilizes
  multiple independent memory states, with a router network directing input
  tokens to specific memory states. This approach greatly enhances the overall
  memory capacity while minimizing memory interference. As a result, MoM performs
  exceptionally well on recall-intensive tasks, surpassing existing linear
  sequence modeling techniques. Despite incorporating multiple memory states, the
  computation of each memory state remains linear in complexity, allowing MoM to
  retain the linear-complexity advantage during training, while
  constant-complexity during inference. Our experimental results show that MoM
  significantly outperforms current linear sequence models on downstream language
  tasks, particularly recall-intensive tasks, and even achieves performance
  comparable to Transformer models. The code is released at
  https://github.com/OpenSparseLLMs/MoM and is also released as a part of
  https://github.com/OpenSparseLLMs/Linear-MoE.
paper_authors._id: 67b6dc1ba7567156c6547880
paper_authors.name: Jusen Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T16:08:01.601Z
paper_authors._id: 65003e857804f04a163328d9
paper_authors.avatarUrl: /avatars/fe32150aabfde8d283b38ccebcf6982e.svg
paper_authors.fullname: Jusen Du
paper_authors.type: user
paper_authors.user: JusenK
paper_authors._id: 67b6dc1ba7567156c6547881
paper_authors.name: Weigao Sun
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-04T08:10:49.466Z
paper_authors._id: 6246bb33da617c00b48e4d92
paper_authors.avatarUrl: /avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
paper_authors.fullname: Weigao Sun
paper_authors.type: user
paper_authors.user: weigao266
paper_authors._id: 67b6dc1ba7567156c6547882
paper_authors.name: Disen Lan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T15:52:54.472Z
paper_authors._id: 66ea643899af9ac3463639b1
paper_authors.avatarUrl: /avatars/252d470e761a57834dee3dbc60dfefed.svg
paper_authors.fullname: Disen Lan
paper_authors.type: user
paper_authors.user: landisen
paper_authors._id: 67b6dc1ba7567156c6547883
paper_authors.name: Jiaxi Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T16:08:08.108Z
paper_authors._id: 665dc35752ff9daa9ba5a4ed
paper_authors.avatarUrl: /avatars/df8b01879d97e599b610fa51414d3a18.svg
paper_authors.fullname: Hu Jiaxi
paper_authors.type: user
paper_authors.user: Jiaxihu2
paper_authors._id: 67b6dc1ba7567156c6547884
paper_authors.name: Yu Cheng
paper_title: MoM: Linear Sequence Modeling with Mixture-of-Memories
paper_summary: Linear sequence modeling methods, such as linear attention, state space
  modeling, and linear RNNs, offer significant efficiency improvements by
  reducing the complexity of training and inference. However, these methods
  typically compress the entire input sequence into a single fixed-size memory
  state, which leads to suboptimal performance on recall-intensive downstream
  tasks. Drawing inspiration from neuroscience, particularly the brain's ability
  to maintain robust long-term memory while mitigating "memory interference", we
  introduce a novel architecture called Mixture-of-Memories (MoM). MoM utilizes
  multiple independent memory states, with a router network directing input
  tokens to specific memory states. This approach greatly enhances the overall
  memory capacity while minimizing memory interference. As a result, MoM performs
  exceptionally well on recall-intensive tasks, surpassing existing linear
  sequence modeling techniques. Despite incorporating multiple memory states, the
  computation of each memory state remains linear in complexity, allowing MoM to
  retain the linear-complexity advantage during training, while
  constant-complexity during inference. Our experimental results show that MoM
  significantly outperforms current linear sequence models on downstream language
  tasks, particularly recall-intensive tasks, and even achieves performance
  comparable to Transformer models. The code is released at
  https://github.com/OpenSparseLLMs/MoM and is also released as a part of
  https://github.com/OpenSparseLLMs/Linear-MoE.
paper_ai_keywords: linear attention state space modeling linear RNNs memory state recall-intensive neuroscience brain's ability long-term memory memory interference Mixture-of-Memories (MoM) router network multiple independent memory states linear-complexity constant-complexity Transformer models