MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
The development of Multimodal Large Language Models (MLLMs) has seen
significant advancements. However, the quantity and quality of multimodal
instruction data have emerged as significant bottlenecks in their progress.
Manually creating multimodal instruction data is both time-consuming and
inefficient, posing challenges in producing instructions of high complexity.
Moreover, distilling instruction data from black-box commercial models (e.g.,
GPT-4o, GPT-4V) often results in simplistic instruction data, which constrains
performance to that of these models. The challenge of curating diverse and
complex instruction data remains substantial. We propose MMEvol, a novel
multimodal instruction data evolution framework that combines fine-grained
perception evolution, cognitive reasoning evolution, and interaction evolution.
This iterative approach breaks through data quality bottlenecks to generate a
complex and diverse image-text instruction dataset, thereby empowering MLLMs
with enhanced capabilities. Beginning with an initial set of instructions,
SEED-163K, we utilize MMEvol to systematically broadens the diversity of
instruction types, integrates reasoning steps to enhance cognitive
capabilities, and extracts detailed information from images to improve visual
understanding and robustness. To comprehensively evaluate the effectiveness of
our data, we train LLaVA-NeXT using the evolved data and conduct experiments
across 13 vision-language tasks. Compared to the baseline trained with seed
data, our approach achieves an average accuracy improvement of 3.1 points and
reaches state-of-the-art (SOTA) performance on 9 of these tasks.
66dfa87056083f2760fa9859
Run Luo
66dfa87056083f2760fa985a
Haonan Zhang
claimed_verified
2024-09-10T07:10:42.161Z
623437bd4f78e3acb8bd14bd
https://cdn-avatars.huggingface.co/v1/production/uploads/623437bd4f78e3acb8bd14bd/6ZIRpE9We4TiMtzDBtPFS.jpeg
Haonan Zhang
user
haonanzhang
66dfa87056083f2760fa985b
Longze Chen
claimed_verified
2024-09-10T07:10:44.103Z
64c7b4d1c547ed5243c07b6c
https://cdn-avatars.huggingface.co/v1/production/uploads/64c7b4d1c547ed5243c07b6c/h96CLBj6dcm01soK2UJzr.jpeg
Longze Chen
user
lzchen2001
66dfa87056083f2760fa985c
Ting-En Lin
claimed_verified
2024-09-10T09:55:56.139Z
62e0ef42edb0462c8d51818d
https://cdn-avatars.huggingface.co/v1/production/uploads/62e0ef42edb0462c8d51818d/3YM7DUynIWiiRFM6_enpg.jpeg
Ting-En Lin
user
tnlin
66dfa87056083f2760fa985d
Xiong Liu
66dfa87056083f2760fa985e
Yuchuan Wu
66dfa87056083f2760fa985f
Min Yang
66dfa87056083f2760fa9860
Minzheng Wang
claimed_verified
2024-09-10T08:07:19.499Z
64bcc373ef8c0e42bf16acc5
/avatars/873308203d28115ae1a9e4d0e26508f4.svg
mz.w
user
iiiiwis
66dfa87056083f2760fa9861
Pengpeng Zeng
66dfa87056083f2760fa9862
Lianli Gao
66dfa87056083f2760fa9863
Heng Tao Shen
66dfa87056083f2760fa9864
Yunshui Li
claimed_verified
2024-09-10T07:10:33.381Z
62e670d33651180f7d334ef3
https://cdn-avatars.huggingface.co/v1/production/uploads/1661829952739-62e670d33651180f7d334ef3.jpeg
Li Yunshui
user
Wa2erGo
66dfa87056083f2760fa9865
Xiaobo Xia
66dfa87056083f2760fa9866
Fei Huang
66dfa87056083f2760fa9867
Jingkuan Song
66dfa87056083f2760fa9868
Yongbin Li
claimed_verified
2024-10-14T10:47:52.545Z
66641b2fd8e1e34bc621e688
https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
Yongbin Li
user
Yongbin-Li
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
The development of Multimodal Large Language Models (MLLMs) has seen
significant advancements. However, the quantity and quality of multimodal
instruction data have emerged as significant bottlenecks in their progress.
Manually creating multimodal instruction data is both time-consuming and
inefficient, posing challenges in producing instructions of high complexity.
Moreover, distilling instruction data from black-box commercial models (e.g.,
GPT-4o, GPT-4V) often results in simplistic instruction data, which constrains
performance to that of these models. The challenge of curating diverse and
complex instruction data remains substantial. We propose MMEvol, a novel
multimodal instruction data evolution framework that combines fine-grained
perception evolution, cognitive reasoning evolution, and interaction evolution.
This iterative approach breaks through data quality bottlenecks to generate a
complex and diverse image-text instruction dataset, thereby empowering MLLMs
with enhanced capabilities. Beginning with an initial set of instructions,
SEED-163K, we utilize MMEvol to systematically broadens the diversity of
instruction types, integrates reasoning steps to enhance cognitive
capabilities, and extracts detailed information from images to improve visual
understanding and robustness. To comprehensively evaluate the effectiveness of
our data, we train LLaVA-NeXT using the evolved data and conduct experiments
across 13 vision-language tasks. Compared to the baseline trained with seed
data, our approach achieves an average accuracy improvement of 3.1 points and
reaches state-of-the-art (SOTA) performance on 9 of these tasks.
MMEvol, a multimodal instruction data evolution framework, enhances the capabilities of Multimodal Large Language Models by generating diverse and complex image-text instruction datasets, leading to improved performance across various vision-language tasks.
Multimodal Large Language Models MMEvol fine-grained perception evolution cognitive reasoning evolution interaction evolution LLaVA-NeXT vision-language tasks state-of-the-art