title: ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance
summary: In this paper, we introduce ILLUME, a unified multimodal large language model
  (MLLM) that seamlessly integrates multimodal understanding and generation
  capabilities within a single large language model through a unified next-token
  prediction formulation. To address the large dataset size typically required
  for image-text alignment, we propose to enhance data efficiency through the
  design of a vision tokenizer that incorporates semantic information and a
  progressive multi-stage training procedure. This approach reduces the dataset
  size to just 15M for pretraining -- over four times fewer than what is
  typically needed -- while achieving competitive or even superior performance
  with existing unified MLLMs, such as Janus. Additionally, to promote
  synergistic enhancement between understanding and generation capabilities,
  which is under-explored in previous works, we introduce a novel self-enhancing
  multimodal alignment scheme. This scheme supervises the MLLM to self-assess the
  consistency between text descriptions and self-generated images, facilitating
  the model to interpret images more accurately and avoid unrealistic and
  incorrect predictions caused by misalignment in image generation. Based on
  extensive experiments, our proposed ILLUME stands out and competes with
  state-of-the-art unified MLLMs and specialized models across various benchmarks
  for multimodal understanding, generation, and editing.
paper_authors._id: 67588d470cdb85095c3f6729
paper_authors.name: Chunwei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:47:08.436Z
paper_authors._id: 67060e1eccd611adc1c56740
paper_authors.avatarUrl: /avatars/936850d0a3f0e4caf55fdaf2937c24ab.svg
paper_authors.fullname: chunwei
paper_authors.type: user
paper_authors.user: chunwei0224
paper_authors._id: 67588d470cdb85095c3f672a
paper_authors.name: Guansong Lu
paper_authors._id: 67588d470cdb85095c3f672b
paper_authors.name: Junwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:48:17.456Z
paper_authors._id: 66a9e095539f1ccd196692b7
paper_authors.avatarUrl: /avatars/9fcdab9c3546bc090a9136ae29de1e4e.svg
paper_authors.fullname: Yang
paper_authors.type: user
paper_authors.user: JunweiYang
paper_authors._id: 67588d470cdb85095c3f672c
paper_authors.name: Runhui Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T14:43:11.526Z
paper_authors._id: 630f0542cc8ed75decb03b68
paper_authors.avatarUrl: /avatars/f76c3603b2700591f33a8a931f7ca664.svg
paper_authors.fullname: huangrh9
paper_authors.type: user
paper_authors.user: huangrh9
paper_authors._id: 67588d470cdb85095c3f672d
paper_authors.name: Jianhua Han
paper_authors._id: 67588d470cdb85095c3f672e
paper_authors.name: Lu Hou
paper_authors._id: 67588d470cdb85095c3f672f
paper_authors.name: Wei Zhang
paper_authors._id: 67588d470cdb85095c3f6730
paper_authors.name: Hang Xu
paper_title: ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance
paper_summary: In this paper, we introduce ILLUME, a unified multimodal large language model
  (MLLM) that seamlessly integrates multimodal understanding and generation
  capabilities within a single large language model through a unified next-token
  prediction formulation. To address the large dataset size typically required
  for image-text alignment, we propose to enhance data efficiency through the
  design of a vision tokenizer that incorporates semantic information and a
  progressive multi-stage training procedure. This approach reduces the dataset
  size to just 15M for pretraining -- over four times fewer than what is
  typically needed -- while achieving competitive or even superior performance
  with existing unified MLLMs, such as Janus. Additionally, to promote
  synergistic enhancement between understanding and generation capabilities,
  which is under-explored in previous works, we introduce a novel self-enhancing
  multimodal alignment scheme. This scheme supervises the MLLM to self-assess the
  consistency between text descriptions and self-generated images, facilitating
  the model to interpret images more accurately and avoid unrealistic and
  incorrect predictions caused by misalignment in image generation. Based on
  extensive experiments, our proposed ILLUME stands out and competes with
  state-of-the-art unified MLLMs and specialized models across various benchmarks
  for multimodal understanding, generation, and editing.
paper_ai_summary: ILLUME, a unified multimodal large language model, integrates multimodal understanding and generation using a vision tokenizer and a self-enhancing alignment scheme, achieving competitive performance with fewer data.
paper_ai_keywords: multimodal large language model unified next-token prediction vision tokenizer semantic information progressive multi-stage training self-enhancing multimodal alignment image-text alignment