MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and
  Instruction-Tuning Dataset for LVLMs
Generating natural and meaningful responses to communicate with multi-modal
human inputs is a fundamental capability of Large Vision-Language
Models(LVLMs). While current open-source LVLMs demonstrate promising
performance in simplified scenarios such as single-turn single-image input,
they fall short in real-world conversation scenarios such as following
instructions in a long context history with multi-turn and multi-images.
Existing LVLM benchmarks primarily focus on single-choice questions or
short-form responses, which do not adequately assess the capabilities of LVLMs
in real-world human-AI interaction applications. Therefore, we introduce MMDU,
a comprehensive benchmark, and MMDU-45k, a large-scale instruction tuning
dataset, designed to evaluate and improve LVLMs' abilities in multi-turn and
multi-image conversations. We employ the clustering algorithm to ffnd the
relevant images and textual descriptions from the open-source Wikipedia and
construct the question-answer pairs by human annotators with the assistance of
the GPT-4o model. MMDU has a maximum of 18k image+text tokens, 20 images, and
27 turns, which is at least 5x longer than previous benchmarks and poses
challenges to current LVLMs. Our in-depth analysis of 15 representative LVLMs
using MMDU reveals that open-source LVLMs lag behind closed-source counterparts
due to limited conversational instruction tuning data. We demonstrate that
ffne-tuning open-source LVLMs on MMDU-45k signiffcantly address this gap,
generating longer and more accurate conversations, and improving scores on MMDU
and existing benchmarks (MMStar: +1.1%, MathVista: +1.5%, ChartQA:+1.2%). Our
contributions pave the way for bridging the gap between current LVLM models and
real-world application demands. This project is available at
https://github.com/Liuziyu77/MMDU.
66712b73f69ec2d1e91bd5e2
Ziyu Liu
admin_assigned
2024-06-18T13:29:39.798Z
643b8846065961b2252bca09
/avatars/5437a8c1aa2180d132375adcb8b969da.svg
Ziyu Liu
user
Objecta
66712b73f69ec2d1e91bd5e3
Tao Chu
66712b73f69ec2d1e91bd5e4
Yuhang Zang
claimed_verified
2024-06-18T13:28:39.907Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
66712b73f69ec2d1e91bd5e5
Xilin Wei
admin_assigned
2024-06-18T13:42:17.007Z
62eb70462f0f5e54df42f778
/avatars/456049dba67638d3cdb330cdf383f272.svg
Xilin Wei
user
Wiselnn
66712b73f69ec2d1e91bd5e6
Xiaoyi Dong
66712b73f69ec2d1e91bd5e7
Pan Zhang
66712b73f69ec2d1e91bd5e8
Zijian Liang
66712b73f69ec2d1e91bd5e9
Yuanjun Xiong
admin_assigned
2024-06-18T13:44:42.288Z
66209605f44be495f123b055
/avatars/b72081a0383a709899f9ba80b4211690.svg
Yuanjun Xiong
user
TheYJ
66712b73f69ec2d1e91bd5ea
Yu Qiao
66712b73f69ec2d1e91bd5eb
Dahua Lin
admin_assigned
2024-06-18T13:43:53.813Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
66712b73f69ec2d1e91bd5ec
Jiaqi Wang
claimed_verified
2024-07-09T07:42:21.812Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and
  Instruction-Tuning Dataset for LVLMs
Generating natural and meaningful responses to communicate with multi-modal
human inputs is a fundamental capability of Large Vision-Language
Models(LVLMs). While current open-source LVLMs demonstrate promising
performance in simplified scenarios such as single-turn single-image input,
they fall short in real-world conversation scenarios such as following
instructions in a long context history with multi-turn and multi-images.
Existing LVLM benchmarks primarily focus on single-choice questions or
short-form responses, which do not adequately assess the capabilities of LVLMs
in real-world human-AI interaction applications. Therefore, we introduce MMDU,
a comprehensive benchmark, and MMDU-45k, a large-scale instruction tuning
dataset, designed to evaluate and improve LVLMs' abilities in multi-turn and
multi-image conversations. We employ the clustering algorithm to ffnd the
relevant images and textual descriptions from the open-source Wikipedia and
construct the question-answer pairs by human annotators with the assistance of
the GPT-4o model. MMDU has a maximum of 18k image+text tokens, 20 images, and
27 turns, which is at least 5x longer than previous benchmarks and poses
challenges to current LVLMs. Our in-depth analysis of 15 representative LVLMs
using MMDU reveals that open-source LVLMs lag behind closed-source counterparts
due to limited conversational instruction tuning data. We demonstrate that
ffne-tuning open-source LVLMs on MMDU-45k signiffcantly address this gap,
generating longer and more accurate conversations, and improving scores on MMDU
and existing benchmarks (MMStar: +1.1%, MathVista: +1.5%, ChartQA:+1.2%). Our
contributions pave the way for bridging the gap between current LVLM models and
real-world application demands. This project is available at
https://github.com/Liuziyu77/MMDU.
A new benchmark (MMDU) and instruction tuning dataset (MMDU-45k) are introduced to evaluate and enhance large vision-language models' performance in multi-turn and multi-image conversations, highlighting the need for improved conversational instruction tuning data.
Large Vision-Language Models LVLMs MMDU MMDU-45k benchmark instruction tuning dataset multi-turn conversations multi-image conversations clustering algorithm Wikipedia human annotators GPT-4o conversational instruction tuning MMStar MathVista ChartQA