title: MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual
    Math Problems?
summary: The remarkable progress of Multi-modal Large Language Models (MLLMs) has
  garnered unparalleled attention, due to their superior performance in visual
  contexts. However, their capabilities in visual math problem-solving remain
  insufficiently evaluated and understood. We investigate current benchmarks to
  incorporate excessive visual content within textual questions, which
  potentially assist MLLMs in deducing answers without truly interpreting the
  input diagrams. To this end, we introduce MathVerse, an all-around visual math
  benchmark designed for an equitable and in-depth evaluation of MLLMs. We
  meticulously collect 2,612 high-quality, multi-subject math problems with
  diagrams from publicly available sources. Each problem is then transformed by
  human annotators into six distinct versions, each offering varying degrees of
  information content in multi-modality, contributing to 15K test samples in
  total. This approach allows MathVerse to comprehensively assess whether and how
  much MLLMs can truly understand the visual diagrams for mathematical reasoning.
  In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a
  fine-grained assessment of the output answers. Rather than naively judging True
  or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and
  then score each step with detailed error analysis, which can reveal the
  intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark
  may provide unique insights to guide the future development of MLLMs. Project
  page: https://mathverse-cuhk.github.io
paper_authors._id: 65fcfd2c7bdbd2233074180f
paper_authors.name: Renrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T08:37:36.971Z
paper_authors._id: 645b8b2687c79b6ec0bb3b7a
paper_authors.avatarUrl: /avatars/00a9db32a42dc950112bf2593bb109cb.svg
paper_authors.fullname: Renrui
paper_authors.type: user
paper_authors.user: ZrrSkywalker
paper_authors._id: 65fcfd2c7bdbd22330741810
paper_authors.name: Dongzhi Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T08:37:39.384Z
paper_authors._id: 6349214f8146350b3a4c5cdf
paper_authors.avatarUrl: /avatars/cfd24caac9a87efb528d0f4c375932bc.svg
paper_authors.fullname: Dongzhi Jiang
paper_authors.type: user
paper_authors.user: CaraJ
paper_authors._id: 65fcfd2c7bdbd22330741811
paper_authors.name: Yichi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-25T09:44:04.582Z
paper_authors._id: 64c9deec0986bd6fa25f366e
paper_authors.avatarUrl: /avatars/d29cde5def2cd7763c0454e2824bdf47.svg
paper_authors.fullname: yichi zhang
paper_authors.type: user
paper_authors.user: void0721
paper_authors._id: 65fcfd2c7bdbd22330741812
paper_authors.name: Haokun Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T09:32:16.407Z
paper_authors._id: 64b899db91876a8d16bed440
paper_authors.avatarUrl: /avatars/cfbc7db053706609c5dc8e0f61ec8a7e.svg
paper_authors.fullname: Haokun Lin
paper_authors.type: user
paper_authors.user: hlface
paper_authors._id: 65fcfd2c7bdbd22330741813
paper_authors.name: Ziyu Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T09:33:08.565Z
paper_authors._id: 647d9ab61a1fcad2fdbf2d3d
paper_authors.avatarUrl: /avatars/48c8aeae8979d2c87df8bde922437d62.svg
paper_authors.fullname: Ziyu Guo
paper_authors.type: user
paper_authors.user: ZiyuG
paper_authors._id: 65fcfd2c7bdbd22330741814
paper_authors.name: Pengshuo Qiu
paper_authors._id: 65fcfd2c7bdbd22330741815
paper_authors.name: Aojun Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T10:05:57.303Z
paper_authors._id: 637de1520d5bb06fbe5207a9
paper_authors.avatarUrl: /avatars/1090851217270c5a858b13e013356d4f.svg
paper_authors.fullname: AJ.Zhou
paper_authors.type: user
paper_authors.user: AJZhou
paper_authors._id: 65fcfd2c7bdbd22330741816
paper_authors.name: Pan Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T09:34:19.182Z
paper_authors._id: 60f5f68fa7fd83d025749234
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60f5f68fa7fd83d025749234/gCeJAZfzaANAcEvI6v5-P.jpeg
paper_authors.fullname: Pan Lu
paper_authors.type: user
paper_authors.user: lupantech
paper_authors._id: 65fcfd2c7bdbd22330741817
paper_authors.name: Kai-Wei Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T09:34:42.634Z
paper_authors._id: 60b7b9d71b90c5d07c23fbd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
paper_authors.fullname: Kai-Wei Chang
paper_authors.type: user
paper_authors.user: kaiweichang
paper_authors._id: 65fcfd2c7bdbd22330741818
paper_authors.name: Peng Gao
paper_authors._id: 65fcfd2c7bdbd22330741819
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T09:35:23.450Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual
    Math Problems?
paper_summary: The remarkable progress of Multi-modal Large Language Models (MLLMs) has
  garnered unparalleled attention, due to their superior performance in visual
  contexts. However, their capabilities in visual math problem-solving remain
  insufficiently evaluated and understood. We investigate current benchmarks to
  incorporate excessive visual content within textual questions, which
  potentially assist MLLMs in deducing answers without truly interpreting the
  input diagrams. To this end, we introduce MathVerse, an all-around visual math
  benchmark designed for an equitable and in-depth evaluation of MLLMs. We
  meticulously collect 2,612 high-quality, multi-subject math problems with
  diagrams from publicly available sources. Each problem is then transformed by
  human annotators into six distinct versions, each offering varying degrees of
  information content in multi-modality, contributing to 15K test samples in
  total. This approach allows MathVerse to comprehensively assess whether and how
  much MLLMs can truly understand the visual diagrams for mathematical reasoning.
  In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a
  fine-grained assessment of the output answers. Rather than naively judging True
  or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and
  then score each step with detailed error analysis, which can reveal the
  intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark
  may provide unique insights to guide the future development of MLLMs. Project
  page: https://mathverse-cuhk.github.io
paper_ai_summary: MathVerse is a comprehensive visual math benchmark with CoT evaluation to assess MLLMs' true understanding and reasoning quality from visual diagrams.
paper_ai_keywords: Multi-modal Large Language Models (MLLMs) MathVerse Chain-of-Thought (CoT) GPT-4(V)