title: MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
summary: The development of Multimodal Large Language Models (MLLMs) has seen
  significant advancements. However, the quantity and quality of multimodal
  instruction data have emerged as significant bottlenecks in their progress.
  Manually creating multimodal instruction data is both time-consuming and
  inefficient, posing challenges in producing instructions of high complexity.
  Moreover, distilling instruction data from black-box commercial models (e.g.,
  GPT-4o, GPT-4V) often results in simplistic instruction data, which constrains
  performance to that of these models. The challenge of curating diverse and
  complex instruction data remains substantial. We propose MMEvol, a novel
  multimodal instruction data evolution framework that combines fine-grained
  perception evolution, cognitive reasoning evolution, and interaction evolution.
  This iterative approach breaks through data quality bottlenecks to generate a
  complex and diverse image-text instruction dataset, thereby empowering MLLMs
  with enhanced capabilities. Beginning with an initial set of instructions,
  SEED-163K, we utilize MMEvol to systematically broadens the diversity of
  instruction types, integrates reasoning steps to enhance cognitive
  capabilities, and extracts detailed information from images to improve visual
  understanding and robustness. To comprehensively evaluate the effectiveness of
  our data, we train LLaVA-NeXT using the evolved data and conduct experiments
  across 13 vision-language tasks. Compared to the baseline trained with seed
  data, our approach achieves an average accuracy improvement of 3.1 points and
  reaches state-of-the-art (SOTA) performance on 9 of these tasks.
paper_authors._id: 66dfa87056083f2760fa9859
paper_authors.name: Run Luo
paper_authors._id: 66dfa87056083f2760fa985a
paper_authors.name: Haonan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T07:10:42.161Z
paper_authors._id: 623437bd4f78e3acb8bd14bd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623437bd4f78e3acb8bd14bd/6ZIRpE9We4TiMtzDBtPFS.jpeg
paper_authors.fullname: Haonan Zhang
paper_authors.type: user
paper_authors.user: haonanzhang
paper_authors._id: 66dfa87056083f2760fa985b
paper_authors.name: Longze Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T07:10:44.103Z
paper_authors._id: 64c7b4d1c547ed5243c07b6c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c7b4d1c547ed5243c07b6c/h96CLBj6dcm01soK2UJzr.jpeg
paper_authors.fullname: Longze Chen
paper_authors.type: user
paper_authors.user: lzchen2001
paper_authors._id: 66dfa87056083f2760fa985c
paper_authors.name: Ting-En Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T09:55:56.139Z
paper_authors._id: 62e0ef42edb0462c8d51818d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e0ef42edb0462c8d51818d/3YM7DUynIWiiRFM6_enpg.jpeg
paper_authors.fullname: Ting-En Lin
paper_authors.type: user
paper_authors.user: tnlin
paper_authors._id: 66dfa87056083f2760fa985d
paper_authors.name: Xiong Liu
paper_authors._id: 66dfa87056083f2760fa985e
paper_authors.name: Yuchuan Wu
paper_authors._id: 66dfa87056083f2760fa985f
paper_authors.name: Min Yang
paper_authors._id: 66dfa87056083f2760fa9860
paper_authors.name: Minzheng Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T08:07:19.499Z
paper_authors._id: 64bcc373ef8c0e42bf16acc5
paper_authors.avatarUrl: /avatars/873308203d28115ae1a9e4d0e26508f4.svg
paper_authors.fullname: mz.w
paper_authors.type: user
paper_authors.user: iiiiwis
paper_authors._id: 66dfa87056083f2760fa9861
paper_authors.name: Pengpeng Zeng
paper_authors._id: 66dfa87056083f2760fa9862
paper_authors.name: Lianli Gao
paper_authors._id: 66dfa87056083f2760fa9863
paper_authors.name: Heng Tao Shen
paper_authors._id: 66dfa87056083f2760fa9864
paper_authors.name: Yunshui Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T07:10:33.381Z
paper_authors._id: 62e670d33651180f7d334ef3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661829952739-62e670d33651180f7d334ef3.jpeg
paper_authors.fullname: LiYunshui
paper_authors.type: user
paper_authors.user: Wa2erGo
paper_authors._id: 66dfa87056083f2760fa9865
paper_authors.name: Xiaobo Xia
paper_authors._id: 66dfa87056083f2760fa9866
paper_authors.name: Fei Huang
paper_authors._id: 66dfa87056083f2760fa9867
paper_authors.name: Jingkuan Song
paper_authors._id: 66dfa87056083f2760fa9868
paper_authors.name: Yongbin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T10:47:52.545Z
paper_authors._id: 66641b2fd8e1e34bc621e688
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
paper_authors.fullname: Yongbin Li
paper_authors.type: user
paper_authors.user: Yongbin-Li
paper_title: MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
paper_summary: The development of Multimodal Large Language Models (MLLMs) has seen
  significant advancements. However, the quantity and quality of multimodal
  instruction data have emerged as significant bottlenecks in their progress.
  Manually creating multimodal instruction data is both time-consuming and
  inefficient, posing challenges in producing instructions of high complexity.
  Moreover, distilling instruction data from black-box commercial models (e.g.,
  GPT-4o, GPT-4V) often results in simplistic instruction data, which constrains
  performance to that of these models. The challenge of curating diverse and
  complex instruction data remains substantial. We propose MMEvol, a novel
  multimodal instruction data evolution framework that combines fine-grained
  perception evolution, cognitive reasoning evolution, and interaction evolution.
  This iterative approach breaks through data quality bottlenecks to generate a
  complex and diverse image-text instruction dataset, thereby empowering MLLMs
  with enhanced capabilities. Beginning with an initial set of instructions,
  SEED-163K, we utilize MMEvol to systematically broadens the diversity of
  instruction types, integrates reasoning steps to enhance cognitive
  capabilities, and extracts detailed information from images to improve visual
  understanding and robustness. To comprehensively evaluate the effectiveness of
  our data, we train LLaVA-NeXT using the evolved data and conduct experiments
  across 13 vision-language tasks. Compared to the baseline trained with seed
  data, our approach achieves an average accuracy improvement of 3.1 points and
  reaches state-of-the-art (SOTA) performance on 9 of these tasks.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) multimodal instruction data denoising autoencoders black-box commercial models (e.g., GPT-4, GPT-4V) simplistic Instruction data fine-grained perception evolution VoN reasoning evolution interaction evolution iterative approach image-text instruction dataset SEED- carefully63K LLaVA-NeXT state-of-the-art (SOTA)