title: SynerGen-VL: Towards Synergistic Image Understanding and Generation with
    Vision Experts and Token Folding
summary: The remarkable success of Large Language Models (LLMs) has extended to the
  multimodal domain, achieving outstanding performance in image understanding and
  generation. Recent efforts to develop unified Multimodal Large Language Models
  (MLLMs) that integrate these capabilities have shown promising results.
  However, existing approaches often involve complex designs in model
  architecture or training pipeline, increasing the difficulty of model training
  and scaling. In this paper, we propose SynerGen-VL, a simple yet powerful
  encoder-free MLLM capable of both image understanding and generation. To
  address challenges identified in existing encoder-free unified MLLMs, we
  introduce the token folding mechanism and the vision-expert-based progressive
  alignment pretraining strategy, which effectively support high-resolution image
  understanding while reducing training complexity. After being trained on
  large-scale mixed image-text data with a unified next-token prediction
  objective, SynerGen-VL achieves or surpasses the performance of existing
  encoder-free unified MLLMs with comparable or smaller parameter sizes, and
  narrows the gap with task-specific state-of-the-art models, highlighting a
  promising path toward future unified MLLMs. Our code and models shall be
  released.
paper_authors._id: 675d657d1c375f21ff6c008f
paper_authors.name: Hao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:41:44.798Z
paper_authors._id: 64b9033777ae61bcc80aa4f3
paper_authors.avatarUrl: /avatars/408c335395c79f3df69fd9bf70abc312.svg
paper_authors.fullname: Hao Li
paper_authors.type: user
paper_authors.user: cpsxhao
paper_authors._id: 675d657d1c375f21ff6c0090
paper_authors.name: Changyao Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:23:50.435Z
paper_authors._id: 64b7475efa7eabaae5f7ba94
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b7475efa7eabaae5f7ba94/YLv44PZM6tw1sxACY1-U_.png
paper_authors.fullname: Changyao Tian
paper_authors.type: user
paper_authors.user: Changyao
paper_authors._id: 675d657d1c375f21ff6c0091
paper_authors.name: Jie Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:24:24.661Z
paper_authors._id: 644a2bccd9a3ae834104b065
paper_authors.avatarUrl: /avatars/ee2caf787796cca438349d10089bdfd1.svg
paper_authors.fullname: Jie Shao
paper_authors.type: user
paper_authors.user: hehesang
paper_authors._id: 675d657d1c375f21ff6c0092
paper_authors.name: Xizhou Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:24:30.898Z
paper_authors._id: 64ae2359179421d320b1694b
paper_authors.avatarUrl: /avatars/c387a75191005bcaa473091de5383a10.svg
paper_authors.fullname: Xizhou Zhu
paper_authors.type: user
paper_authors.user: Einsiedler
paper_authors._id: 675d657d1c375f21ff6c0093
paper_authors.name: Zhaokai Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:41:42.930Z
paper_authors._id: 665d4b515fdfe8f923e347a7
paper_authors.avatarUrl: /avatars/d114b24c02dadfca0a8aee104755a8ec.svg
paper_authors.fullname: Zhaokai Wang
paper_authors.type: user
paper_authors.user: wzk1015
paper_authors._id: 675d657d1c375f21ff6c0094
paper_authors.name: Jinguo Zhu
paper_authors._id: 675d657d1c375f21ff6c0095
paper_authors.name: Wenhan Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:24:50.436Z
paper_authors._id: 66efe658de163a536aa84178
paper_authors.avatarUrl: /avatars/fddc42450cabf41ca1ab2f70b185f51c.svg
paper_authors.fullname: dou wenhan
paper_authors.type: user
paper_authors.user: douwh
paper_authors._id: 675d657d1c375f21ff6c0096
paper_authors.name: Xiaogang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:25:01.045Z
paper_authors._id: 66149831f9e58fe02b08f10e
paper_authors.avatarUrl: /avatars/493636f2368e689c9241e53bc2e7b97e.svg
paper_authors.fullname: wangxiaogang
paper_authors.type: user
paper_authors.user: wangxiaogang
paper_authors._id: 675d657d1c375f21ff6c0097
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:25:17.271Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 675d657d1c375f21ff6c0098
paper_authors.name: Lewei Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:25:24.403Z
paper_authors._id: 65ead3ea908526a39082e641
paper_authors.avatarUrl: /avatars/dcf870695fd56b06ca03d82f831e9019.svg
paper_authors.fullname: Lewei Lu
paper_authors.type: user
paper_authors.user: luotto
paper_authors._id: 675d657d1c375f21ff6c0099
paper_authors.name: Jifeng Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:25:31.520Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_title: SynerGen-VL: Towards Synergistic Image Understanding and Generation with
    Vision Experts and Token Folding
paper_summary: The remarkable success of Large Language Models (LLMs) has extended to the
  multimodal domain, achieving outstanding performance in image understanding and
  generation. Recent efforts to develop unified Multimodal Large Language Models
  (MLLMs) that integrate these capabilities have shown promising results.
  However, existing approaches often involve complex designs in model
  architecture or training pipeline, increasing the difficulty of model training
  and scaling. In this paper, we propose SynerGen-VL, a simple yet powerful
  encoder-free MLLM capable of both image understanding and generation. To
  address challenges identified in existing encoder-free unified MLLMs, we
  introduce the token folding mechanism and the vision-expert-based progressive
  alignment pretraining strategy, which effectively support high-resolution image
  understanding while reducing training complexity. After being trained on
  large-scale mixed image-text data with a unified next-token prediction
  objective, SynerGen-VL achieves or surpasses the performance of existing
  encoder-free unified MLLMs with comparable or smaller parameter sizes, and
  narrows the gap with task-specific state-of-the-art models, highlighting a
  promising path toward future unified MLLMs. Our code and models shall be
  released.
paper_ai_summary: SynerGen-VL, an encoder-free MLLM using token folding and vision-expert-based progressive alignment, performs comparably to task-specific models on image understanding and generation with reduced training complexity.
paper_ai_keywords: Large Language Models Multimodal Large Language Models SynerGen-VL token folding mechanism vision-expert-based progressive alignment pretraining strategy high-resolution image understanding unified next-token prediction objective