title: Multimodal Mamba: Decoder-only Multimodal State Space Model via
    Quadratic to Linear Distillation
summary: Recent Multimodal Large Language Models (MLLMs) have achieved remarkable
  performance but face deployment challenges due to their quadratic computational
  complexity, growing Key-Value cache requirements, and reliance on separate
  vision encoders. We propose mmMamba, a framework for developing
  linear-complexity native multimodal state space models through progressive
  distillation from existing MLLMs using moderate academic computational
  resources. Our approach enables the direct conversion of trained decoder-only
  MLLMs to linear-complexity architectures without requiring pre-trained
  RNN-based LLM or vision encoders. We propose an seeding strategy to carve Mamba
  from trained Transformer and a three-stage distillation recipe, which can
  effectively transfer the knowledge from Transformer to Mamba while preserving
  multimodal capabilities. Our method also supports flexible hybrid architectures
  that combine Transformer and Mamba layers for customizable
  efficiency-performance trade-offs. Distilled from the Transformer-based
  decoder-only HoVLE, mmMamba-linear achieves competitive performance against
  existing linear and quadratic-complexity VLMs, while mmMamba-hybrid further
  improves performance significantly, approaching HoVLE's capabilities. At 103K
  tokens, mmMamba-linear demonstrates 20.6times speedup and 75.8% GPU memory
  reduction compared to HoVLE, while mmMamba-hybrid achieves 13.5times speedup
  and 60.2% memory savings. Code and models are released at
  https://github.com/hustvl/mmMamba
paper_authors._id: 67b54b04bd51b4e46e39d287
paper_authors.name: Bencheng Liao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:01:00.934Z
paper_authors._id: 6577073fc2bf55b1f6bafb49
paper_authors.avatarUrl: /avatars/58803398b1a918b7570db17893e65122.svg
paper_authors.fullname: liao
paper_authors.type: user
paper_authors.user: LegendBC
paper_authors._id: 67b54b04bd51b4e46e39d288
paper_authors.name: Hongyuan Tao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:13:43.349Z
paper_authors._id: 66a105bb456284adf458d656
paper_authors.avatarUrl: /avatars/b543a324f7e159d6e84bc68915e93d24.svg
paper_authors.fullname: Tao Hongyuan
paper_authors.type: user
paper_authors.user: HongyuanTao
paper_authors._id: 67b54b04bd51b4e46e39d289
paper_authors.name: Qian Zhang
paper_authors._id: 67b54b04bd51b4e46e39d28a
paper_authors.name: Tianheng Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:00:58.351Z
paper_authors._id: 646b3db131968a60a01e4cf5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646b3db131968a60a01e4cf5/DhfdqUYQaD1Qa8Svw996J.jpeg
paper_authors.fullname: Tianheng Cheng
paper_authors.type: user
paper_authors.user: wondervictor
paper_authors._id: 67b54b04bd51b4e46e39d28b
paper_authors.name: Yingyue Li
paper_authors._id: 67b54b04bd51b4e46e39d28c
paper_authors.name: Haoran Yin
paper_authors._id: 67b54b04bd51b4e46e39d28d
paper_authors.name: Wenyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:12:36.866Z
paper_authors._id: 66c2e7fc934e2f07753542ac
paper_authors.avatarUrl: /avatars/f6fa3f94435cf1c1d06daa6c925d07d0.svg
paper_authors.fullname: LWY
paper_authors.type: user
paper_authors.user: wenyuliu
paper_authors._id: 67b54b04bd51b4e46e39d28e
paper_authors.name: Xinggang Wang
paper_title: Multimodal Mamba: Decoder-only Multimodal State Space Model via
    Quadratic to Linear Distillation
paper_summary: Recent Multimodal Large Language Models (MLLMs) have achieved remarkable
  performance but face deployment challenges due to their quadratic computational
  complexity, growing Key-Value cache requirements, and reliance on separate
  vision encoders. We propose mmMamba, a framework for developing
  linear-complexity native multimodal state space models through progressive
  distillation from existing MLLMs using moderate academic computational
  resources. Our approach enables the direct conversion of trained decoder-only
  MLLMs to linear-complexity architectures without requiring pre-trained
  RNN-based LLM or vision encoders. We propose an seeding strategy to carve Mamba
  from trained Transformer and a three-stage distillation recipe, which can
  effectively transfer the knowledge from Transformer to Mamba while preserving
  multimodal capabilities. Our method also supports flexible hybrid architectures
  that combine Transformer and Mamba layers for customizable
  efficiency-performance trade-offs. Distilled from the Transformer-based
  decoder-only HoVLE, mmMamba-linear achieves competitive performance against
  existing linear and quadratic-complexity VLMs, while mmMamba-hybrid further
  improves performance significantly, approaching HoVLE's capabilities. At 103K
  tokens, mmMamba-linear demonstrates 20.6times speedup and 75.8% GPU memory
  reduction compared to HoVLE, while mmMamba-hybrid achieves 13.5times speedup
  and 60.2% memory savings. Code and models are released at
  https://github.com/hustvl/mmMamba
paper_ai_keywords: Multimodal Large Language Models (MLLMs) Key-Value cache vision encoders linear-complexity environment multimodal state space models progressive distillation decoder-only MLLMs RNN-based LLM vision encoders Schw seeding strategy Mamba Transformer three-stage distillation recipe multimodal capabilities flexible hybrid architectures Transformer and Mamba layers efficiency-performance trade-offs HoVLE VLMs mmMamba-linear mmMamba-hybrid