title: Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large
    Language Models
summary: Large language models (LLMs) have demonstrated impressive reasoning
  capabilities, particularly in textual mathematical problem-solving. However,
  existing open-source image instruction fine-tuning datasets, containing limited
  question-answer pairs per image, do not fully exploit visual information to
  enhance the multimodal mathematical reasoning capabilities of Multimodal LLMs
  (MLLMs). To bridge this gap, we address the lack of high-quality, diverse
  multimodal mathematical datasets by collecting 40K high-quality images with
  question-answer pairs from 24 existing datasets and synthesizing 320K new
  pairs, creating the MathV360K dataset, which enhances both the breadth and
  depth of multimodal mathematical questions. We introduce Math-LLaVA, a
  LLaVA-1.5-based model fine-tuned with MathV360K. This novel approach
  significantly improves the multimodal mathematical reasoning capabilities of
  LLaVA-1.5, achieving a 19-point increase and comparable performance to GPT-4V
  on MathVista's minitest split. Furthermore, Math-LLaVA demonstrates enhanced
  generalizability, showing substantial improvements on the MMMU benchmark. Our
  research highlights the importance of dataset diversity and synthesis in
  advancing MLLMs' mathematical reasoning abilities. The code and data are
  available at: https://github.com/HZQ950419/Math-LLaVA.
paper_authors._id: 667c06bf1976894a8bdac826
paper_authors.name: Wenhao Shi
paper_authors._id: 667c06bf1976894a8bdac827
paper_authors.name: Zhiqiang Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:03:10.563Z
paper_authors._id: 637f228152229c63921119c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f228152229c63921119c3/acwXorra1r9_7i3KlBFjS.jpeg
paper_authors.fullname: Zhiqiang Hu
paper_authors.type: user
paper_authors.user: Zhiqiang007
paper_authors._id: 667c06bf1976894a8bdac828
paper_authors.name: Yi Bin
paper_authors._id: 667c06bf1976894a8bdac829
paper_authors.name: Junhua Liu
paper_authors._id: 667c06bf1976894a8bdac82a
paper_authors.name: Yang Yang
paper_authors._id: 667c06bf1976894a8bdac82b
paper_authors.name: See-Kiong Ng
paper_authors._id: 667c06bf1976894a8bdac82c
paper_authors.name: Lidong Bing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T15:58:21.608Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_authors._id: 667c06bf1976894a8bdac82d
paper_authors.name: Roy Ka-Wei Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T15:58:15.887Z
paper_authors._id: 64b94cc0e3d41dbd6974ae45
paper_authors.avatarUrl: /avatars/5edb9d9465addfceccef04c4465a34e6.svg
paper_authors.fullname: Roy Ka-Wei lee
paper_authors.type: user
paper_authors.user: sroylee
paper_title: Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large
    Language Models
paper_summary: Large language models (LLMs) have demonstrated impressive reasoning
  capabilities, particularly in textual mathematical problem-solving. However,
  existing open-source image instruction fine-tuning datasets, containing limited
  question-answer pairs per image, do not fully exploit visual information to
  enhance the multimodal mathematical reasoning capabilities of Multimodal LLMs
  (MLLMs). To bridge this gap, we address the lack of high-quality, diverse
  multimodal mathematical datasets by collecting 40K high-quality images with
  question-answer pairs from 24 existing datasets and synthesizing 320K new
  pairs, creating the MathV360K dataset, which enhances both the breadth and
  depth of multimodal mathematical questions. We introduce Math-LLaVA, a
  LLaVA-1.5-based model fine-tuned with MathV360K. This novel approach
  significantly improves the multimodal mathematical reasoning capabilities of
  LLaVA-1.5, achieving a 19-point increase and comparable performance to GPT-4V
  on MathVista's minitest split. Furthermore, Math-LLaVA demonstrates enhanced
  generalizability, showing substantial improvements on the MMMU benchmark. Our
  research highlights the importance of dataset diversity and synthesis in
  advancing MLLMs' mathematical reasoning abilities. The code and data are
  available at: https://github.com/HZQ950419/Math-LLaVA.
paper_ai_keywords: large language models (LLMs) multimodal LLMs (MLLMs) image instruction fine-tuning question-answer pairs visual information multimodal mathematical reasoning MathV360K dataset Math-LLaVA aVA-1.5 MathVista's minitest split MMMU benchmark dataset diversity dataset synthesis