Multimodal Mamba: Decoder-only Multimodal State Space Model via
  Quadratic to Linear Distillation
Recent Multimodal Large Language Models (MLLMs) have achieved remarkable
performance but face deployment challenges due to their quadratic computational
complexity, growing Key-Value cache requirements, and reliance on separate
vision encoders. We propose mmMamba, a framework for developing
linear-complexity native multimodal state space models through progressive
distillation from existing MLLMs using moderate academic computational
resources. Our approach enables the direct conversion of trained decoder-only
MLLMs to linear-complexity architectures without requiring pre-trained
RNN-based LLM or vision encoders. We propose an seeding strategy to carve Mamba
from trained Transformer and a three-stage distillation recipe, which can
effectively transfer the knowledge from Transformer to Mamba while preserving
multimodal capabilities. Our method also supports flexible hybrid architectures
that combine Transformer and Mamba layers for customizable
efficiency-performance trade-offs. Distilled from the Transformer-based
decoder-only HoVLE, mmMamba-linear achieves competitive performance against
existing linear and quadratic-complexity VLMs, while mmMamba-hybrid further
improves performance significantly, approaching HoVLE's capabilities. At 103K
tokens, mmMamba-linear demonstrates 20.6times speedup and 75.8% GPU memory
reduction compared to HoVLE, while mmMamba-hybrid achieves 13.5times speedup
and 60.2% memory savings. Code and models are released at
https://github.com/hustvl/mmMamba
67b54b04bd51b4e46e39d287
Bencheng Liao
claimed_verified
2025-02-19T09:01:00.934Z
6577073fc2bf55b1f6bafb49
/avatars/58803398b1a918b7570db17893e65122.svg
Bencheng liao
user
LegendBC
67b54b04bd51b4e46e39d288
Hongyuan Tao
admin_assigned
2025-02-19T10:13:43.349Z
66a105bb456284adf458d656
/avatars/b543a324f7e159d6e84bc68915e93d24.svg
Tao Hongyuan
user
HongyuanTao
67b54b04bd51b4e46e39d289
Qian Zhang
67b54b04bd51b4e46e39d28a
Tianheng Cheng
claimed_verified
2025-02-19T09:00:58.351Z
646b3db131968a60a01e4cf5
https://cdn-avatars.huggingface.co/v1/production/uploads/646b3db131968a60a01e4cf5/DhfdqUYQaD1Qa8Svw996J.jpeg
Tianheng Cheng
user
wondervictor
67b54b04bd51b4e46e39d28b
Yingyue Li
67b54b04bd51b4e46e39d28c
Haoran Yin
67b54b04bd51b4e46e39d28d
Wenyu Liu
admin_assigned
2025-02-19T10:12:36.866Z
66c2e7fc934e2f07753542ac
/avatars/f6fa3f94435cf1c1d06daa6c925d07d0.svg
LWY
user
wenyuliu
67b54b04bd51b4e46e39d28e
Xinggang Wang
Multimodal Mamba: Decoder-only Multimodal State Space Model via
  Quadratic to Linear Distillation
Recent Multimodal Large Language Models (MLLMs) have achieved remarkable
performance but face deployment challenges due to their quadratic computational
complexity, growing Key-Value cache requirements, and reliance on separate
vision encoders. We propose mmMamba, a framework for developing
linear-complexity native multimodal state space models through progressive
distillation from existing MLLMs using moderate academic computational
resources. Our approach enables the direct conversion of trained decoder-only
MLLMs to linear-complexity architectures without requiring pre-trained
RNN-based LLM or vision encoders. We propose an seeding strategy to carve Mamba
from trained Transformer and a three-stage distillation recipe, which can
effectively transfer the knowledge from Transformer to Mamba while preserving
multimodal capabilities. Our method also supports flexible hybrid architectures
that combine Transformer and Mamba layers for customizable
efficiency-performance trade-offs. Distilled from the Transformer-based
decoder-only HoVLE, mmMamba-linear achieves competitive performance against
existing linear and quadratic-complexity VLMs, while mmMamba-hybrid further
improves performance significantly, approaching HoVLE's capabilities. At 103K
tokens, mmMamba-linear demonstrates 20.6times speedup and 75.8% GPU memory
reduction compared to HoVLE, while mmMamba-hybrid achieves 13.5times speedup
and 60.2% memory savings. Code and models are released at
https://github.com/hustvl/mmMamba
A framework converts existing decoder-only Multimodal Large Language Models to linear-complexity models using distillation, reducing computational complexity and memory usage while maintaining performance.
Multimodal Large Language Models MLMLs mmMamba linear-complexity multimodal state space models progressive distillation decoder-only MLLMs Transformer seeding strategy distillation recipe multimodal capabilities hybrid architectures HoVLE VLMs speedup GPU memory reduction