ARMOR v0.1: Empowering Autoregressive Multimodal Understanding Model
  with Interleaved Multimodal Generation via Asymmetric Synergy
Unified models (UniMs) for multimodal understanding and generation have
recently received much attention in the area of vision and language. Existing
UniMs are designed to simultaneously learn both multimodal understanding and
generation capabilities, demanding substantial computational resources, and
often struggle to generate interleaved text-image. We present ARMOR, a
resource-efficient and pure autoregressive framework that achieves both
understanding and generation by fine-tuning existing multimodal large language
models (MLLMs). Specifically, ARMOR extends existing MLLMs from three
perspectives: (1) For model architecture, an asymmetric encoder-decoder
architecture with a forward-switching mechanism is introduced to unify
embedding space integrating textual and visual modalities for enabling natural
text-image interleaved generation with minimal computational overhead. (2) For
training data, a meticulously curated, high-quality interleaved dataset is
collected for fine-tuning MLLMs. (3) For the training algorithm, we propose a
``what or how to generate" algorithm to empower existing MLLMs with multimodal
generation capabilities while preserving their multimodal understanding
capabilities, through three progressive training stages based on the collected
dataset. Experimental results demonstrate that ARMOR upgrades existing MLLMs to
UniMs with promising image generation capabilities, using limited training
resources. Our code will be released soon at https://armor.github.io.
67d7e4ec1414fcb6196e79ba
Jianwen Sun
67d7e4ec1414fcb6196e79bb
Yukang Feng
admin_assigned
2025-03-17T14:45:01.498Z
66d94f2a36aa5055694dfe04
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/grAN83brH0E4_S0__yLdv.jpeg
fengyukang
user
finyorko
67d7e4ec1414fcb6196e79bc
Chuanhao Li
claimed_verified
2025-06-23T08:33:49.457Z
674ea59a8f2e7614a6c72f26
/avatars/86fd4c6d7d33435de49b35659bf65265.svg
Chuanhao
user
ChuanhaoLi
67d7e4ec1414fcb6196e79bd
Fanrui Zhang
admin_assigned
2025-03-17T14:44:41.770Z
665305eff0c8c891cae7fe01
/avatars/1f372e3bc6a4eb19ef702ec96a391c96.svg
Fanrui Zhang
user
fanrui00
67d7e4ec1414fcb6196e79be
Zizhen Li
67d7e4ec1414fcb6196e79bf
Jiaxin Ai
67d7e4ec1414fcb6196e79c0
Sizhuo Zhou
67d7e4ec1414fcb6196e79c1
Yu Dai
67d7e4ec1414fcb6196e79c2
Shenglin Zhang
admin_assigned
2025-03-17T14:45:56.321Z
6674d02914e2aebef893779e
/avatars/acdbe3820462b87126c8f1e14f0d1a60.svg
ZhangShenglin
user
ZhangShenglin
67d7e4ec1414fcb6196e79c3
Kaipeng Zhang
admin_assigned
2025-03-17T14:44:19.968Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
ARMOR v0.1: Empowering Autoregressive Multimodal Understanding Model
  with Interleaved Multimodal Generation via Asymmetric Synergy
Unified models (UniMs) for multimodal understanding and generation have
recently received much attention in the area of vision and language. Existing
UniMs are designed to simultaneously learn both multimodal understanding and
generation capabilities, demanding substantial computational resources, and
often struggle to generate interleaved text-image. We present ARMOR, a
resource-efficient and pure autoregressive framework that achieves both
understanding and generation by fine-tuning existing multimodal large language
models (MLLMs). Specifically, ARMOR extends existing MLLMs from three
perspectives: (1) For model architecture, an asymmetric encoder-decoder
architecture with a forward-switching mechanism is introduced to unify
embedding space integrating textual and visual modalities for enabling natural
text-image interleaved generation with minimal computational overhead. (2) For
training data, a meticulously curated, high-quality interleaved dataset is
collected for fine-tuning MLLMs. (3) For the training algorithm, we propose a
``what or how to generate" algorithm to empower existing MLLMs with multimodal
generation capabilities while preserving their multimodal understanding
capabilities, through three progressive training stages based on the collected
dataset. Experimental results demonstrate that ARMOR upgrades existing MLLMs to
UniMs with promising image generation capabilities, using limited training
resources. Our code will be released soon at https://armor.github.io.
ARMOR is a resource-efficient, autoregressive framework that enhances existing multimodal large language models for both understanding and interleaved text-image generation.
Unified models UniMs multimodal understanding generation capabilities multimodal large language models MLLMs asymmetric encoder-decoder forward-switching mechanism embedding space natural text-image interleaved generation high-quality interleaved dataset what or how to generate algorithm multimodal generation multimodal understanding