title: MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and
    Instruction-Tuning Dataset for LVLMs
summary: Generating natural and meaningful responses to communicate with multi-modal
  human inputs is a fundamental capability of Large Vision-Language
  Models(LVLMs). While current open-source LVLMs demonstrate promising
  performance in simplified scenarios such as single-turn single-image input,
  they fall short in real-world conversation scenarios such as following
  instructions in a long context history with multi-turn and multi-images.
  Existing LVLM benchmarks primarily focus on single-choice questions or
  short-form responses, which do not adequately assess the capabilities of LVLMs
  in real-world human-AI interaction applications. Therefore, we introduce MMDU,
  a comprehensive benchmark, and MMDU-45k, a large-scale instruction tuning
  dataset, designed to evaluate and improve LVLMs' abilities in multi-turn and
  multi-image conversations. We employ the clustering algorithm to ffnd the
  relevant images and textual descriptions from the open-source Wikipedia and
  construct the question-answer pairs by human annotators with the assistance of
  the GPT-4o model. MMDU has a maximum of 18k image+text tokens, 20 images, and
  27 turns, which is at least 5x longer than previous benchmarks and poses
  challenges to current LVLMs. Our in-depth analysis of 15 representative LVLMs
  using MMDU reveals that open-source LVLMs lag behind closed-source counterparts
  due to limited conversational instruction tuning data. We demonstrate that
  ffne-tuning open-source LVLMs on MMDU-45k signiffcantly address this gap,
  generating longer and more accurate conversations, and improving scores on MMDU
  and existing benchmarks (MMStar: +1.1%, MathVista: +1.5%, ChartQA:+1.2%). Our
  contributions pave the way for bridging the gap between current LVLM models and
  real-world application demands. This project is available at
  https://github.com/Liuziyu77/MMDU.
paper_authors._id: 66712b73f69ec2d1e91bd5e2
paper_authors.name: Ziyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T13:29:39.798Z
paper_authors._id: 643b8846065961b2252bca09
paper_authors.avatarUrl: /avatars/5437a8c1aa2180d132375adcb8b969da.svg
paper_authors.fullname: Ziyu Liu
paper_authors.type: user
paper_authors.user: Objecta
paper_authors._id: 66712b73f69ec2d1e91bd5e3
paper_authors.name: Tao Chu
paper_authors._id: 66712b73f69ec2d1e91bd5e4
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T13:28:39.907Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/iUQm5FAomzqYi6fkqIn9F.jpeg
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 66712b73f69ec2d1e91bd5e5
paper_authors.name: Xilin Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T13:42:17.007Z
paper_authors._id: 62eb70462f0f5e54df42f778
paper_authors.avatarUrl: /avatars/456049dba67638d3cdb330cdf383f272.svg
paper_authors.fullname: Xilin Wei
paper_authors.type: user
paper_authors.user: Wiselnn
paper_authors._id: 66712b73f69ec2d1e91bd5e6
paper_authors.name: Xiaoyi Dong
paper_authors._id: 66712b73f69ec2d1e91bd5e7
paper_authors.name: Pan Zhang
paper_authors._id: 66712b73f69ec2d1e91bd5e8
paper_authors.name: Zijian Liang
paper_authors._id: 66712b73f69ec2d1e91bd5e9
paper_authors.name: Yuanjun Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T13:44:42.288Z
paper_authors._id: 66209605f44be495f123b055
paper_authors.avatarUrl: /avatars/b72081a0383a709899f9ba80b4211690.svg
paper_authors.fullname: Yuanjun Xiong
paper_authors.type: user
paper_authors.user: TheYJ
paper_authors._id: 66712b73f69ec2d1e91bd5ea
paper_authors.name: Yu Qiao
paper_authors._id: 66712b73f69ec2d1e91bd5eb
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T13:43:53.813Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 66712b73f69ec2d1e91bd5ec
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T07:42:21.812Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_title: MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and
    Instruction-Tuning Dataset for LVLMs
paper_summary: Generating natural and meaningful responses to communicate with multi-modal
  human inputs is a fundamental capability of Large Vision-Language
  Models(LVLMs). While current open-source LVLMs demonstrate promising
  performance in simplified scenarios such as single-turn single-image input,
  they fall short in real-world conversation scenarios such as following
  instructions in a long context history with multi-turn and multi-images.
  Existing LVLM benchmarks primarily focus on single-choice questions or
  short-form responses, which do not adequately assess the capabilities of LVLMs
  in real-world human-AI interaction applications. Therefore, we introduce MMDU,
  a comprehensive benchmark, and MMDU-45k, a large-scale instruction tuning
  dataset, designed to evaluate and improve LVLMs' abilities in multi-turn and
  multi-image conversations. We employ the clustering algorithm to ffnd the
  relevant images and textual descriptions from the open-source Wikipedia and
  construct the question-answer pairs by human annotators with the assistance of
  the GPT-4o model. MMDU has a maximum of 18k image+text tokens, 20 images, and
  27 turns, which is at least 5x longer than previous benchmarks and poses
  challenges to current LVLMs. Our in-depth analysis of 15 representative LVLMs
  using MMDU reveals that open-source LVLMs lag behind closed-source counterparts
  due to limited conversational instruction tuning data. We demonstrate that
  ffne-tuning open-source LVLMs on MMDU-45k signiffcantly address this gap,
  generating longer and more accurate conversations, and improving scores on MMDU
  and existing benchmarks (MMStar: +1.1%, MathVista: +1.5%, ChartQA:+1.2%). Our
  contributions pave the way for bridging the gap between current LVLM models and
  real-world application demands. This project is available at
  https://github.com/Liuziyu77/MMDU.
paper_ai_keywords: Large Vision-Language Models (LVLMs) multi-modal human inputs single-turn single-image input real-world conversation scenarios long context history multi-turn multi-images single-choice questions short-form responses MMDU MMDU-45k instruction tuning dataset clustering algorithm Wikipedia human annotators GPT-4o model image+text tokens in-depth analysis closed-source counterparts conversational instruction tuning data fine-tuning MMStar MathVista ChartQA