title: ReMamba: Equip Mamba with Effective Long-Sequence Modeling
summary: While the Mamba architecture demonstrates superior inference efficiency and
  competitive performance on short-context natural language processing (NLP)
  tasks, empirical evidence suggests its capacity to comprehend long contexts is
  limited compared to transformer-based models. In this study, we investigate the
  long-context efficiency issues of the Mamba models and propose ReMamba, which
  enhances Mamba's ability to comprehend long contexts. ReMamba incorporates
  selective compression and adaptation techniques within a two-stage re-forward
  process, incurring minimal additional inference costs overhead. Experimental
  results on the LongBench and L-Eval benchmarks demonstrate ReMamba's efficacy,
  improving over the baselines by 3.2 and 1.6 points, respectively, and attaining
  performance almost on par with same-size transformer models.
paper_authors._id: 66cfe43588a57d59e6cabe59
paper_authors.name: Danlong Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T08:00:54.734Z
paper_authors._id: 6567ea516b1e4d61d8e16e46
paper_authors.avatarUrl: /avatars/ff33037da2a5f35377e1854e77b5f1ff.svg
paper_authors.fullname: Yuan Danlong
paper_authors.type: user
paper_authors.user: lblankl
paper_authors._id: 66cfe43588a57d59e6cabe5a
paper_authors.name: Jiahao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T08:57:42.071Z
paper_authors._id: 64b75e6e9ebb7e6c7dd5f7c5
paper_authors.avatarUrl: /avatars/5173c10cce77c429802e51e3a28368df.svg
paper_authors.fullname: glaicer liu
paper_authors.type: user
paper_authors.user: glacier000
paper_authors._id: 66cfe43588a57d59e6cabe5b
paper_authors.name: Bei Li
paper_authors._id: 66cfe43588a57d59e6cabe5c
paper_authors.name: Huishuai Zhang
paper_authors._id: 66cfe43588a57d59e6cabe5d
paper_authors.name: Jingang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T08:01:34.650Z
paper_authors._id: 647097cbcfd57849518e656b
paper_authors.avatarUrl: /avatars/c66fe0add29c1bde9e3a98bf4a8793b9.svg
paper_authors.fullname: Jingang Wang
paper_authors.type: user
paper_authors.user: bitwjg
paper_authors._id: 66cfe43588a57d59e6cabe5e
paper_authors.name: Xunliang Cai
paper_authors._id: 66cfe43588a57d59e6cabe5f
paper_authors.name: Dongyan Zhao
paper_title: ReMamba: Equip Mamba with Effective Long-Sequence Modeling
paper_summary: While the Mamba architecture demonstrates superior inference efficiency and
  competitive performance on short-context natural language processing (NLP)
  tasks, empirical evidence suggests its capacity to comprehend long contexts is
  limited compared to transformer-based models. In this study, we investigate the
  long-context efficiency issues of the Mamba models and propose ReMamba, which
  enhances Mamba's ability to comprehend long contexts. ReMamba incorporates
  selective compression and adaptation techniques within a two-stage re-forward
  process, incurring minimal additional inference costs overhead. Experimental
  results on the LongBench and L-Eval benchmarks demonstrate ReMamba's efficacy,
  improving over the baselines by 3.2 and 1.6 points, respectively, and attaining
  performance almost on par with same-size transformer models.
paper_ai_keywords: inference efficiency natural language processing (NLP) transformer-based models long-context efficiency selective compression re-forward process overhead LongBench L-Eval performance