ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and
  Diffusion Refinement
We present ILLUME+ that leverages dual visual tokenization and a diffusion
decoder to improve both deep semantic understanding and high-fidelity image
generation. Existing unified models have struggled to simultaneously handle the
three fundamental capabilities in a unified model: understanding, generation,
and editing. Models like Chameleon and EMU3 utilize VQGAN for image
discretization, due to the lack of deep semantic interaction, they lag behind
specialist models like LLaVA in visual understanding tasks. To mitigate this,
LaViT and ILLUME employ semantic encoders for tokenization, but they struggle
with image editing due to poor texture preservation. Meanwhile, Janus series
decouples the input and output image representation, limiting their abilities
to seamlessly handle interleaved image-text understanding and generation. In
contrast, ILLUME+ introduces a unified dual visual tokenizer, DualViTok, which
preserves both fine-grained textures and text-aligned semantics while enabling
a coarse-to-fine image representation strategy for multimodal understanding and
generation. Additionally, we employ a diffusion model as the image detokenizer
for enhanced generation quality and efficient super-resolution. ILLUME+ follows
a continuous-input, discrete-output scheme within the unified MLLM and adopts a
progressive training procedure that supports dynamic resolution across the
vision tokenizer, MLLM, and diffusion decoder. This design allows for flexible
and efficient context-aware image editing and generation across diverse tasks.
ILLUME+ (3B) exhibits competitive performance against existing unified MLLMs
and specialized models across multimodal understanding, generation, and editing
benchmarks. With its strong performance, ILLUME+ provides a scalable and
versatile foundation for future multimodal applications. Project Page:
https://illume-unified-mllm.github.io/.
67edfe07f5d1509d1a990178
Runhui Huang
claimed_verified
2025-04-03T13:49:29.196Z
630f0542cc8ed75decb03b68
/avatars/f76c3603b2700591f33a8a931f7ca664.svg
huangrh9
user
huangrh9
67edfe07f5d1509d1a990179
Chunwei Wang
admin_assigned
2025-04-03T14:01:13.452Z
644131e5666947ad561349c1
/avatars/8a2d9be81cef30cb3bc71a695c802d2f.svg
Wangchunwei
user
17day
67edfe07f5d1509d1a99017a
Junwei Yang
67edfe07f5d1509d1a99017b
Guansong Lu
67edfe07f5d1509d1a99017c
Yunlong Yuan
67edfe07f5d1509d1a99017d
Jianhua Han
admin_assigned
2025-04-03T14:02:30.946Z
6357c8afa43d8d1bbe64efc9
/avatars/901a656e7327d5d949f10812f35f19b8.svg
Jianhua Han
user
hanjianhua44
67edfe07f5d1509d1a99017e
Lu Hou
admin_assigned
2025-04-03T14:02:38.473Z
5f86882fee5616341bc51da1
/avatars/b0c5883d6253ef67639d1cea591fa337.svg
Lu Hou
user
houlu369
67edfe07f5d1509d1a99017f
Wei Zhang
67edfe07f5d1509d1a990180
Lanqing Hong
admin_assigned
2025-04-03T14:02:44.952Z
671207b95550f581fcce3cfd
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wqvv-QQ59dNT1KSpOUQLa.png
Lanqing HONG
user
racheltechie
67edfe07f5d1509d1a990181
Hengshuang Zhao
67edfe07f5d1509d1a990182
Hang Xu
ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and
  Diffusion Refinement
We present ILLUME+ that leverages dual visual tokenization and a diffusion
decoder to improve both deep semantic understanding and high-fidelity image
generation. Existing unified models have struggled to simultaneously handle the
three fundamental capabilities in a unified model: understanding, generation,
and editing. Models like Chameleon and EMU3 utilize VQGAN for image
discretization, due to the lack of deep semantic interaction, they lag behind
specialist models like LLaVA in visual understanding tasks. To mitigate this,
LaViT and ILLUME employ semantic encoders for tokenization, but they struggle
with image editing due to poor texture preservation. Meanwhile, Janus series
decouples the input and output image representation, limiting their abilities
to seamlessly handle interleaved image-text understanding and generation. In
contrast, ILLUME+ introduces a unified dual visual tokenizer, DualViTok, which
preserves both fine-grained textures and text-aligned semantics while enabling
a coarse-to-fine image representation strategy for multimodal understanding and
generation. Additionally, we employ a diffusion model as the image detokenizer
for enhanced generation quality and efficient super-resolution. ILLUME+ follows
a continuous-input, discrete-output scheme within the unified MLLM and adopts a
progressive training procedure that supports dynamic resolution across the
vision tokenizer, MLLM, and diffusion decoder. This design allows for flexible
and efficient context-aware image editing and generation across diverse tasks.
ILLUME+ (3B) exhibits competitive performance against existing unified MLLMs
and specialized models across multimodal understanding, generation, and editing
benchmarks. With its strong performance, ILLUME+ provides a scalable and
versatile foundation for future multimodal applications. Project Page:
https://illume-unified-mllm.github.io/.
ILLUME+ enhances multimodal understanding, generation, and editing by introducing a dual visual tokenizer that preserves textures and semantics, and using a diffusion decoder for high-quality image generation.
dual visual tokenization diffusion decoder deep semantic understanding high-fidelity image generation unified models VQGAN LaViT semantic encoders DualViTok coarse-to-fine image representation multimodal understanding generation image editing texture preservation continuous-input discrete-output scheme progressive training dynamic resolution