title: ARMOR v0.1: Empowering Autoregressive Multimodal Understanding Model
    with Interleaved Multimodal Generation via Asymmetric Synergy
summary: Unified models (UniMs) for multimodal understanding and generation have
  recently received much attention in the area of vision and language. Existing
  UniMs are designed to simultaneously learn both multimodal understanding and
  generation capabilities, demanding substantial computational resources, and
  often struggle to generate interleaved text-image. We present ARMOR, a
  resource-efficient and pure autoregressive framework that achieves both
  understanding and generation by fine-tuning existing multimodal large language
  models (MLLMs). Specifically, ARMOR extends existing MLLMs from three
  perspectives: (1) For model architecture, an asymmetric encoder-decoder
  architecture with a forward-switching mechanism is introduced to unify
  embedding space integrating textual and visual modalities for enabling natural
  text-image interleaved generation with minimal computational overhead. (2) For
  training data, a meticulously curated, high-quality interleaved dataset is
  collected for fine-tuning MLLMs. (3) For the training algorithm, we propose a
  ``what or how to generate" algorithm to empower existing MLLMs with multimodal
  generation capabilities while preserving their multimodal understanding
  capabilities, through three progressive training stages based on the collected
  dataset. Experimental results demonstrate that ARMOR upgrades existing MLLMs to
  UniMs with promising image generation capabilities, using limited training
  resources. Our code will be released soon at https://armor.github.io.
paper_authors._id: 67d7e4ec1414fcb6196e79ba
paper_authors.name: Jianwen Sun
paper_authors._id: 67d7e4ec1414fcb6196e79bb
paper_authors.name: Yukang Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:45:01.498Z
paper_authors._id: 66d94f2a36aa5055694dfe04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/grAN83brH0E4_S0__yLdv.jpeg
paper_authors.fullname: fengyukang
paper_authors.type: user
paper_authors.user: finyorko
paper_authors._id: 67d7e4ec1414fcb6196e79bc
paper_authors.name: Chuanhao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:45:07.745Z
paper_authors._id: 6533f7ecb3852ed1ceb48e47
paper_authors.avatarUrl: /avatars/5d767c093e73f06a89f625c3a5903902.svg
paper_authors.fullname: Chuanhao Li
paper_authors.type: user
paper_authors.user: cyrilli
paper_authors._id: 67d7e4ec1414fcb6196e79bd
paper_authors.name: Fanrui Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:44:41.770Z
paper_authors._id: 665305eff0c8c891cae7fe01
paper_authors.avatarUrl: /avatars/1f372e3bc6a4eb19ef702ec96a391c96.svg
paper_authors.fullname: Fanrui Zhang
paper_authors.type: user
paper_authors.user: fanrui00
paper_authors._id: 67d7e4ec1414fcb6196e79be
paper_authors.name: Zizhen Li
paper_authors._id: 67d7e4ec1414fcb6196e79bf
paper_authors.name: Jiaxin Ai
paper_authors._id: 67d7e4ec1414fcb6196e79c0
paper_authors.name: Sizhuo Zhou
paper_authors._id: 67d7e4ec1414fcb6196e79c1
paper_authors.name: Yu Dai
paper_authors._id: 67d7e4ec1414fcb6196e79c2
paper_authors.name: Shenglin Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:45:56.321Z
paper_authors._id: 6674d02914e2aebef893779e
paper_authors.avatarUrl: /avatars/acdbe3820462b87126c8f1e14f0d1a60.svg
paper_authors.fullname: ZhangShenglin
paper_authors.type: user
paper_authors.user: ZhangShenglin
paper_authors._id: 67d7e4ec1414fcb6196e79c3
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:44:19.968Z
paper_authors._id: 65f1713552c38a91e0a445e8
paper_authors.avatarUrl: /avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
paper_authors.fullname: kaipeng
paper_authors.type: user
paper_authors.user: kpzhang996
paper_title: ARMOR v0.1: Empowering Autoregressive Multimodal Understanding Model
    with Interleaved Multimodal Generation via Asymmetric Synergy
paper_summary: Unified models (UniMs) for multimodal understanding and generation have
  recently received much attention in the area of vision and language. Existing
  UniMs are designed to simultaneously learn both multimodal understanding and
  generation capabilities, demanding substantial computational resources, and
  often struggle to generate interleaved text-image. We present ARMOR, a
  resource-efficient and pure autoregressive framework that achieves both
  understanding and generation by fine-tuning existing multimodal large language
  models (MLLMs). Specifically, ARMOR extends existing MLLMs from three
  perspectives: (1) For model architecture, an asymmetric encoder-decoder
  architecture with a forward-switching mechanism is introduced to unify
  embedding space integrating textual and visual modalities for enabling natural
  text-image interleaved generation with minimal computational overhead. (2) For
  training data, a meticulously curated, high-quality interleaved dataset is
  collected for fine-tuning MLLMs. (3) For the training algorithm, we propose a
  ``what or how to generate" algorithm to empower existing MLLMs with multimodal
  generation capabilities while preserving their multimodal understanding
  capabilities, through three progressive training stages based on the collected
  dataset. Experimental results demonstrate that ARMOR upgrades existing MLLMs to
  UniMs with promising image generation capabilities, using limited training
  resources. Our code will be released soon at https://armor.github.io.
paper_ai_keywords: asymmetric encoder-decoder architecture forward-switching mechanism embedding space textual modality visual modality natural text-image interleaved generation computational overhead high-quality interleaved dataset multimodal large language models ``what or how to generate" algorithm progressive training stages image generation capabilities multimodal understanding capabilities