title: MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at
    Scale
summary: Open-source multimodal large language models (MLLMs) have shown significant
  potential in a broad range of multimodal tasks. However, their reasoning
  capabilities remain constrained by existing instruction-tuning datasets, which
  were predominately repurposed from academic datasets such as VQA, AI2D, and
  ChartQA. These datasets target simplistic tasks, and only provide phrase-level
  answers without any intermediate rationales. To address these challenges, we
  introduce a scalable and cost-effective method to construct a large-scale
  multimodal instruction-tuning dataset with rich intermediate rationales
  designed to elicit CoT reasoning. Using only open models, we create a dataset
  containing 12M instruction-response pairs to cover diverse, reasoning-intensive
  tasks with detailed and faithful rationales. Experiments demonstrate that
  training MLLMs on this dataset significantly improves reasoning capabilities,
  achieving state-of-the-art performance on benchmarks such as MathVerse (+8.1%),
  MMMU-Pro (+7%), and MuirBench (+13.3%). Additionally, the model demonstrates
  notable improvements of up to 4% on non-reasoning-based benchmarks. Ablation
  studies further highlight the importance of key components, such as rewriting
  and self-filtering, in the dataset construction process.
paper_authors._id: 6756592b59dd4e7470378f6c
paper_authors.name: Jarvis Guo
paper_authors._id: 6756592b59dd4e7470378f6d
paper_authors.name: Tuney Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T13:26:09.219Z
paper_authors._id: 64ab99dcb76bfd863eba64c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
paper_authors.fullname: TY.Zheng
paper_authors.type: user
paper_authors.user: aaabiao
paper_authors._id: 6756592b59dd4e7470378f6e
paper_authors.name: Yuelin Bai
paper_authors._id: 6756592b59dd4e7470378f6f
paper_authors.name: Bo Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:05:45.993Z
paper_authors._id: 62d3f7d84b0933c48f3cdd9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: luodian
paper_authors._id: 6756592b59dd4e7470378f70
paper_authors.name: Yubo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T08:41:23.329Z
paper_authors._id: 636a35eff8d9af4aea181608
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
paper_authors.fullname: yubo
paper_authors.type: user
paper_authors.user: ubowang
paper_authors._id: 6756592b59dd4e7470378f71
paper_authors.name: King Zhu
paper_authors._id: 6756592b59dd4e7470378f72
paper_authors.name: Yizhi Li
paper_authors._id: 6756592b59dd4e7470378f73
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T14:59:33.210Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_authors._id: 6756592b59dd4e7470378f74
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T14:59:38.971Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_authors._id: 6756592b59dd4e7470378f75
paper_authors.name: Xiang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:06:01.577Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_title: MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at
    Scale
paper_summary: Open-source multimodal large language models (MLLMs) have shown significant
  potential in a broad range of multimodal tasks. However, their reasoning
  capabilities remain constrained by existing instruction-tuning datasets, which
  were predominately repurposed from academic datasets such as VQA, AI2D, and
  ChartQA. These datasets target simplistic tasks, and only provide phrase-level
  answers without any intermediate rationales. To address these challenges, we
  introduce a scalable and cost-effective method to construct a large-scale
  multimodal instruction-tuning dataset with rich intermediate rationales
  designed to elicit CoT reasoning. Using only open models, we create a dataset
  containing 12M instruction-response pairs to cover diverse, reasoning-intensive
  tasks with detailed and faithful rationales. Experiments demonstrate that
  training MLLMs on this dataset significantly improves reasoning capabilities,
  achieving state-of-the-art performance on benchmarks such as MathVerse (+8.1%),
  MMMU-Pro (+7%), and MuirBench (+13.3%). Additionally, the model demonstrates
  notable improvements of up to 4% on non-reasoning-based benchmarks. Ablation
  studies further highlight the importance of key components, such as rewriting
  and self-filtering, in the dataset construction process.
paper_ai_summary: A large-scale multimodal instruction-tuning dataset with intermediate rationales improves reasoning capabilities in large language models, achieving state-of-the-art performance across benchmarks.
paper_ai_keywords: multimodal large language models instruction-tuning dataset intermediate rationales CoT reasoning MathVerse MMMU-Pro MuirBench rewriting self-filtering