title: OmniMamba: Efficient and Unified Multimodal Understanding and Generation
    via State Space Models
summary: Recent advancements in unified multimodal understanding and visual generation
  (or multimodal generation) models have been hindered by their quadratic
  computational complexity and dependence on large-scale training data. We
  present OmniMamba, the first linear-architecture-based multimodal generation
  model that generates both text and images through a unified next-token
  prediction paradigm. The model fully leverages Mamba-2's high computational and
  memory efficiency, extending its capabilities from text generation to
  multimodal generation. To address the data inefficiency of existing unified
  models, we propose two key innovations: (1) decoupled vocabularies to guide
  modality-specific generation, and (2) task-specific LoRA for
  parameter-efficient adaptation. Furthermore, we introduce a decoupled two-stage
  training strategy to mitigate data imbalance between two tasks. Equipped with
  these techniques, OmniMamba achieves competitive performance with JanusFlow
  while surpassing Show-o across benchmarks, despite being trained on merely 2M
  image-text pairs, which is 1,000 times fewer than Show-o. Notably, OmniMamba
  stands out with outstanding inference efficiency, achieving up to a 119.2 times
  speedup and 63% GPU memory reduction for long-sequence generation compared to
  Transformer-based counterparts. Code and models are released at
  https://github.com/hustvl/OmniMamba
paper_authors._id: 67d0f892a189f3978638e154
paper_authors.name: Jialv Zou
paper_authors._id: 67d0f892a189f3978638e155
paper_authors.name: Bencheng Liao
paper_authors._id: 67d0f892a189f3978638e156
paper_authors.name: Qian Zhang
paper_authors._id: 67d0f892a189f3978638e157
paper_authors.name: Wenyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-12T15:55:12.045Z
paper_authors._id: 66c2e7fc934e2f07753542ac
paper_authors.avatarUrl: /avatars/f6fa3f94435cf1c1d06daa6c925d07d0.svg
paper_authors.fullname: LWY
paper_authors.type: user
paper_authors.user: wenyuliu
paper_authors._id: 67d0f892a189f3978638e158
paper_authors.name: Xinggang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-12T15:55:00.983Z
paper_authors._id: 62600de6d47e3dbae32ce1ce
paper_authors.avatarUrl: /avatars/a536417cfec6e10ac415091bd1829426.svg
paper_authors.fullname: Xinggang Wang
paper_authors.type: user
paper_authors.user: xinggangw
paper_title: OmniMamba: Efficient and Unified Multimodal Understanding and Generation
    via State Space Models
paper_summary: Recent advancements in unified multimodal understanding and visual generation
  (or multimodal generation) models have been hindered by their quadratic
  computational complexity and dependence on large-scale training data. We
  present OmniMamba, the first linear-architecture-based multimodal generation
  model that generates both text and images through a unified next-token
  prediction paradigm. The model fully leverages Mamba-2's high computational and
  memory efficiency, extending its capabilities from text generation to
  multimodal generation. To address the data inefficiency of existing unified
  models, we propose two key innovations: (1) decoupled vocabularies to guide
  modality-specific generation, and (2) task-specific LoRA for
  parameter-efficient adaptation. Furthermore, we introduce a decoupled two-stage
  training strategy to mitigate data imbalance between two tasks. Equipped with
  these techniques, OmniMamba achieves competitive performance with JanusFlow
  while surpassing Show-o across benchmarks, despite being trained on merely 2M
  image-text pairs, which is 1,000 times fewer than Show-o. Notably, OmniMamba
  stands out with outstanding inference efficiency, achieving up to a 119.2 times
  speedup and 63% GPU memory reduction for long-sequence generation compared to
  Transformer-based counterparts. Code and models are released at
  https://github.com/hustvl/OmniMamba
paper_ai_keywords: OmniMamba linear-architecture-based multimodal generation model unified next-token prediction paradigm Mamba-2 computational efficiency memory efficiency decoupled vocabularies task-specific LoRA parameter-efficient adaptation decoupled two-stage training strategy data imbalance Show-o benchmark inference efficiency Transformer-based counterparts