title: ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and
    Diffusion Refinement
summary: We present ILLUME+ that leverages dual visual tokenization and a diffusion
  decoder to improve both deep semantic understanding and high-fidelity image
  generation. Existing unified models have struggled to simultaneously handle the
  three fundamental capabilities in a unified model: understanding, generation,
  and editing. Models like Chameleon and EMU3 utilize VQGAN for image
  discretization, due to the lack of deep semantic interaction, they lag behind
  specialist models like LLaVA in visual understanding tasks. To mitigate this,
  LaViT and ILLUME employ semantic encoders for tokenization, but they struggle
  with image editing due to poor texture preservation. Meanwhile, Janus series
  decouples the input and output image representation, limiting their abilities
  to seamlessly handle interleaved image-text understanding and generation. In
  contrast, ILLUME+ introduces a unified dual visual tokenizer, DualViTok, which
  preserves both fine-grained textures and text-aligned semantics while enabling
  a coarse-to-fine image representation strategy for multimodal understanding and
  generation. Additionally, we employ a diffusion model as the image detokenizer
  for enhanced generation quality and efficient super-resolution. ILLUME+ follows
  a continuous-input, discrete-output scheme within the unified MLLM and adopts a
  progressive training procedure that supports dynamic resolution across the
  vision tokenizer, MLLM, and diffusion decoder. This design allows for flexible
  and efficient context-aware image editing and generation across diverse tasks.
  ILLUME+ (3B) exhibits competitive performance against existing unified MLLMs
  and specialized models across multimodal understanding, generation, and editing
  benchmarks. With its strong performance, ILLUME+ provides a scalable and
  versatile foundation for future multimodal applications. Project Page:
  https://illume-unified-mllm.github.io/.
paper_authors._id: 67edfe07f5d1509d1a990178
paper_authors.name: Runhui Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T13:49:29.196Z
paper_authors._id: 630f0542cc8ed75decb03b68
paper_authors.avatarUrl: /avatars/f76c3603b2700591f33a8a931f7ca664.svg
paper_authors.fullname: huangrh9
paper_authors.type: user
paper_authors.user: huangrh9
paper_authors._id: 67edfe07f5d1509d1a990179
paper_authors.name: Chunwei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T14:01:13.452Z
paper_authors._id: 644131e5666947ad561349c1
paper_authors.avatarUrl: /avatars/8a2d9be81cef30cb3bc71a695c802d2f.svg
paper_authors.fullname: Wangchunwei
paper_authors.type: user
paper_authors.user: 17day
paper_authors._id: 67edfe07f5d1509d1a99017a
paper_authors.name: Junwei Yang
paper_authors._id: 67edfe07f5d1509d1a99017b
paper_authors.name: Guansong Lu
paper_authors._id: 67edfe07f5d1509d1a99017c
paper_authors.name: Yunlong Yuan
paper_authors._id: 67edfe07f5d1509d1a99017d
paper_authors.name: Jianhua Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T14:02:30.946Z
paper_authors._id: 6357c8afa43d8d1bbe64efc9
paper_authors.avatarUrl: /avatars/901a656e7327d5d949f10812f35f19b8.svg
paper_authors.fullname: Jianhua Han
paper_authors.type: user
paper_authors.user: hanjianhua44
paper_authors._id: 67edfe07f5d1509d1a99017e
paper_authors.name: Lu Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T14:02:38.473Z
paper_authors._id: 5f86882fee5616341bc51da1
paper_authors.avatarUrl: /avatars/b0c5883d6253ef67639d1cea591fa337.svg
paper_authors.fullname: Lu Hou
paper_authors.type: user
paper_authors.user: houlu369
paper_authors._id: 67edfe07f5d1509d1a99017f
paper_authors.name: Wei Zhang
paper_authors._id: 67edfe07f5d1509d1a990180
paper_authors.name: Lanqing Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T14:02:44.952Z
paper_authors._id: 671207b95550f581fcce3cfd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wqvv-QQ59dNT1KSpOUQLa.png
paper_authors.fullname: Lanqing HONG
paper_authors.type: user
paper_authors.user: racheltechie
paper_authors._id: 67edfe07f5d1509d1a990181
paper_authors.name: Hengshuang Zhao
paper_authors._id: 67edfe07f5d1509d1a990182
paper_authors.name: Hang Xu
paper_title: ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and
    Diffusion Refinement
paper_summary: We present ILLUME+ that leverages dual visual tokenization and a diffusion
  decoder to improve both deep semantic understanding and high-fidelity image
  generation. Existing unified models have struggled to simultaneously handle the
  three fundamental capabilities in a unified model: understanding, generation,
  and editing. Models like Chameleon and EMU3 utilize VQGAN for image
  discretization, due to the lack of deep semantic interaction, they lag behind
  specialist models like LLaVA in visual understanding tasks. To mitigate this,
  LaViT and ILLUME employ semantic encoders for tokenization, but they struggle
  with image editing due to poor texture preservation. Meanwhile, Janus series
  decouples the input and output image representation, limiting their abilities
  to seamlessly handle interleaved image-text understanding and generation. In
  contrast, ILLUME+ introduces a unified dual visual tokenizer, DualViTok, which
  preserves both fine-grained textures and text-aligned semantics while enabling
  a coarse-to-fine image representation strategy for multimodal understanding and
  generation. Additionally, we employ a diffusion model as the image detokenizer
  for enhanced generation quality and efficient super-resolution. ILLUME+ follows
  a continuous-input, discrete-output scheme within the unified MLLM and adopts a
  progressive training procedure that supports dynamic resolution across the
  vision tokenizer, MLLM, and diffusion decoder. This design allows for flexible
  and efficient context-aware image editing and generation across diverse tasks.
  ILLUME+ (3B) exhibits competitive performance against existing unified MLLMs
  and specialized models across multimodal understanding, generation, and editing
  benchmarks. With its strong performance, ILLUME+ provides a scalable and
  versatile foundation for future multimodal applications. Project Page:
  https://illume-unified-mllm.github.io/.
paper_ai_summary: ILLUME+ enhances multimodal understanding, generation, and editing by introducing a dual visual tokenizer that preserves textures and semantics, and using a diffusion decoder for high-quality image generation.
paper_ai_keywords: dual visual tokenization diffusion decoder deep semantic understanding high-fidelity image generation unified models VQGAN LaViT semantic encoders DualViTok coarse-to-fine image representation multimodal understanding generation image editing texture preservation continuous-input discrete-output scheme progressive training dynamic resolution