ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance
In this paper, we introduce ILLUME, a unified multimodal large language model
(MLLM) that seamlessly integrates multimodal understanding and generation
capabilities within a single large language model through a unified next-token
prediction formulation. To address the large dataset size typically required
for image-text alignment, we propose to enhance data efficiency through the
design of a vision tokenizer that incorporates semantic information and a
progressive multi-stage training procedure. This approach reduces the dataset
size to just 15M for pretraining -- over four times fewer than what is
typically needed -- while achieving competitive or even superior performance
with existing unified MLLMs, such as Janus. Additionally, to promote
synergistic enhancement between understanding and generation capabilities,
which is under-explored in previous works, we introduce a novel self-enhancing
multimodal alignment scheme. This scheme supervises the MLLM to self-assess the
consistency between text descriptions and self-generated images, facilitating
the model to interpret images more accurately and avoid unrealistic and
incorrect predictions caused by misalignment in image generation. Based on
extensive experiments, our proposed ILLUME stands out and competes with
state-of-the-art unified MLLMs and specialized models across various benchmarks
for multimodal understanding, generation, and editing.
67588d470cdb85095c3f6729
Chunwei Wang
admin_assigned
2024-12-11T13:47:08.436Z
67060e1eccd611adc1c56740
/avatars/936850d0a3f0e4caf55fdaf2937c24ab.svg
chunwei
user
chunwei0224
67588d470cdb85095c3f672a
Guansong Lu
67588d470cdb85095c3f672b
Junwei Yang
admin_assigned
2024-12-11T13:48:17.456Z
66a9e095539f1ccd196692b7
/avatars/9fcdab9c3546bc090a9136ae29de1e4e.svg
Yang
user
JunweiYang
67588d470cdb85095c3f672c
Runhui Huang
claimed_verified
2025-02-21T14:43:11.526Z
630f0542cc8ed75decb03b68
/avatars/f76c3603b2700591f33a8a931f7ca664.svg
huangrh9
user
huangrh9
67588d470cdb85095c3f672d
Jianhua Han
67588d470cdb85095c3f672e
Lu Hou
67588d470cdb85095c3f672f
Wei Zhang
67588d470cdb85095c3f6730
Hang Xu
ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance
In this paper, we introduce ILLUME, a unified multimodal large language model
(MLLM) that seamlessly integrates multimodal understanding and generation
capabilities within a single large language model through a unified next-token
prediction formulation. To address the large dataset size typically required
for image-text alignment, we propose to enhance data efficiency through the
design of a vision tokenizer that incorporates semantic information and a
progressive multi-stage training procedure. This approach reduces the dataset
size to just 15M for pretraining -- over four times fewer than what is
typically needed -- while achieving competitive or even superior performance
with existing unified MLLMs, such as Janus. Additionally, to promote
synergistic enhancement between understanding and generation capabilities,
which is under-explored in previous works, we introduce a novel self-enhancing
multimodal alignment scheme. This scheme supervises the MLLM to self-assess the
consistency between text descriptions and self-generated images, facilitating
the model to interpret images more accurately and avoid unrealistic and
incorrect predictions caused by misalignment in image generation. Based on
extensive experiments, our proposed ILLUME stands out and competes with
state-of-the-art unified MLLMs and specialized models across various benchmarks
for multimodal understanding, generation, and editing.
ILLUME, a unified multimodal large language model, integrates multimodal understanding and generation using a vision tokenizer and a self-enhancing alignment scheme, achieving competitive performance with fewer data.
multimodal large language model unified next-token prediction vision tokenizer semantic information progressive multi-stage training self-enhancing multimodal alignment image-text alignment