title: TransMamba: Flexibly Switching between Transformer and Mamba
summary: Transformers are the cornerstone of modern large language models, but their
  quadratic computational complexity limits efficiency in long-sequence
  processing. Recent advancements in Mamba, a state space model (SSM) with linear
  complexity, offer promising efficiency gains but suffer from unstable
  contextual learning and multitask generalization. This paper proposes
  TransMamba, a novel framework that unifies Transformer and Mamba through shared
  parameter matrices (e.g., QKV and CBx), and thus could dynamically switch
  between attention and SSM mechanisms at different token lengths and layers. We
  design the Memory converter to bridge Transformer and Mamba by converting
  attention outputs into SSM-compatible states, ensuring seamless information
  flow at TransPoints where the transformation happens. The TransPoint scheduling
  is also thoroughly explored for further improvements. We conducted extensive
  experiments demonstrating that TransMamba achieves superior training efficiency
  and performance compared to baselines, and validated the deeper consistency
  between Transformer and Mamba paradigms, offering a scalable solution for
  next-generation sequence modeling.
paper_authors._id: 67ecb89d0210bff02fd3591e
paper_authors.name: Yixing Li
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-02T04:10:06.783Z
paper_authors._id: 66b4e3d850b87d84498bbc89
paper_authors.avatarUrl: /avatars/526c0cabf3a2c019a13bb82fcc7b43e9.svg
paper_authors.fullname: YixingLi
paper_authors.type: user
paper_authors.user: Yixinglee
paper_authors._id: 67ecb89d0210bff02fd3591f
paper_authors.name: Ruobing Xie
paper_authors._id: 67ecb89d0210bff02fd35920
paper_authors.name: Zhen Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:21:39.891Z
paper_authors._id: 62c4057732fa66fedacca0db
paper_authors.avatarUrl: /avatars/813e8c9fcd14dde1fdd415408f61bec2.svg
paper_authors.fullname: Zhen Yang
paper_authors.type: user
paper_authors.user: andyyang
paper_authors._id: 67ecb89d0210bff02fd35921
paper_authors.name: Xingwu Sun
paper_authors._id: 67ecb89d0210bff02fd35922
paper_authors.name: Shuaipeng Li
paper_authors._id: 67ecb89d0210bff02fd35923
paper_authors.name: Weidong Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-13T19:27:30.584Z
paper_authors._id: 615c2f8416284d83200e1105
paper_authors.avatarUrl: /avatars/8e0c89925f714de58f0fc77265acd76f.svg
paper_authors.fullname: han weidong
paper_authors.type: user
paper_authors.user: dongdong2021
paper_authors._id: 67ecb89d0210bff02fd35924
paper_authors.name: Zhanhui Kang
paper_authors._id: 67ecb89d0210bff02fd35925
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:51:32.687Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_authors._id: 67ecb89d0210bff02fd35926
paper_authors.name: Chengzhong Xu
paper_authors._id: 67ecb89d0210bff02fd35927
paper_authors.name: Di Wang
paper_authors._id: 67ecb89d0210bff02fd35928
paper_authors.name: Jie Jiang
paper_title: TransMamba: Flexibly Switching between Transformer and Mamba
paper_summary: Transformers are the cornerstone of modern large language models, but their
  quadratic computational complexity limits efficiency in long-sequence
  processing. Recent advancements in Mamba, a state space model (SSM) with linear
  complexity, offer promising efficiency gains but suffer from unstable
  contextual learning and multitask generalization. This paper proposes
  TransMamba, a novel framework that unifies Transformer and Mamba through shared
  parameter matrices (e.g., QKV and CBx), and thus could dynamically switch
  between attention and SSM mechanisms at different token lengths and layers. We
  design the Memory converter to bridge Transformer and Mamba by converting
  attention outputs into SSM-compatible states, ensuring seamless information
  flow at TransPoints where the transformation happens. The TransPoint scheduling
  is also thoroughly explored for further improvements. We conducted extensive
  experiments demonstrating that TransMamba achieves superior training efficiency
  and performance compared to baselines, and validated the deeper consistency
  between Transformer and Mamba paradigms, offering a scalable solution for
  next-generation sequence modeling.
paper_ai_summary: TransMamba combines Transformers and Mamba frameworks to leverage the strengths of both, achieving efficient and effective long-sequence processing through shared parameters and dynamic switching mechanisms.
paper_ai_keywords: Transformers Mamba state space model shared parameter matrices QKV CBx attention mechanisms Memory converter TransPoints TransPoint scheduling sequence modeling