MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual
  Math Problems?
The remarkable progress of Multi-modal Large Language Models (MLLMs) has
garnered unparalleled attention, due to their superior performance in visual
contexts. However, their capabilities in visual math problem-solving remain
insufficiently evaluated and understood. We investigate current benchmarks to
incorporate excessive visual content within textual questions, which
potentially assist MLLMs in deducing answers without truly interpreting the
input diagrams. To this end, we introduce MathVerse, an all-around visual math
benchmark designed for an equitable and in-depth evaluation of MLLMs. We
meticulously collect 2,612 high-quality, multi-subject math problems with
diagrams from publicly available sources. Each problem is then transformed by
human annotators into six distinct versions, each offering varying degrees of
information content in multi-modality, contributing to 15K test samples in
total. This approach allows MathVerse to comprehensively assess whether and how
much MLLMs can truly understand the visual diagrams for mathematical reasoning.
In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a
fine-grained assessment of the output answers. Rather than naively judging True
or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and
then score each step with detailed error analysis, which can reveal the
intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark
may provide unique insights to guide the future development of MLLMs. Project
page: https://mathverse-cuhk.github.io
65fcfd2c7bdbd2233074180f
Renrui Zhang
claimed_verified
2024-03-22T08:37:36.971Z
645b8b2687c79b6ec0bb3b7a
/avatars/00a9db32a42dc950112bf2593bb109cb.svg
Renrui
user
ZrrSkywalker
65fcfd2c7bdbd22330741810
Dongzhi Jiang
claimed_verified
2024-03-22T08:37:39.384Z
6349214f8146350b3a4c5cdf
/avatars/cfd24caac9a87efb528d0f4c375932bc.svg
Dongzhi Jiang
user
CaraJ
65fcfd2c7bdbd22330741811
Yichi Zhang
claimed_verified
2024-03-25T09:44:04.582Z
64c9deec0986bd6fa25f366e
/avatars/d29cde5def2cd7763c0454e2824bdf47.svg
yichi zhang
user
void0721
65fcfd2c7bdbd22330741812
Haokun Lin
admin_assigned
2024-03-22T09:32:16.407Z
64b899db91876a8d16bed440
/avatars/cfbc7db053706609c5dc8e0f61ec8a7e.svg
Haokun Lin
user
hlface
65fcfd2c7bdbd22330741813
Ziyu Guo
admin_assigned
2024-03-22T09:33:08.565Z
647d9ab61a1fcad2fdbf2d3d
/avatars/48c8aeae8979d2c87df8bde922437d62.svg
Ziyu Guo
user
ZiyuG
65fcfd2c7bdbd22330741814
Pengshuo Qiu
65fcfd2c7bdbd22330741815
Aojun Zhou
claimed_verified
2024-03-22T10:05:57.303Z
637de1520d5bb06fbe5207a9
/avatars/1090851217270c5a858b13e013356d4f.svg
AJ.Zhou
user
AJZhou
65fcfd2c7bdbd22330741816
Pan Lu
admin_assigned
2024-03-22T09:34:19.182Z
60f5f68fa7fd83d025749234
https://cdn-avatars.huggingface.co/v1/production/uploads/60f5f68fa7fd83d025749234/gCeJAZfzaANAcEvI6v5-P.jpeg
Pan Lu
user
lupantech
65fcfd2c7bdbd22330741817
Kai-Wei Chang
admin_assigned
2024-03-22T09:34:42.634Z
60b7b9d71b90c5d07c23fbd0
https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
Kai-Wei Chang
user
kaiweichang
65fcfd2c7bdbd22330741818
Peng Gao
65fcfd2c7bdbd22330741819
Hongsheng Li
admin_assigned
2024-03-22T09:35:23.450Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual
  Math Problems?
The remarkable progress of Multi-modal Large Language Models (MLLMs) has
garnered unparalleled attention, due to their superior performance in visual
contexts. However, their capabilities in visual math problem-solving remain
insufficiently evaluated and understood. We investigate current benchmarks to
incorporate excessive visual content within textual questions, which
potentially assist MLLMs in deducing answers without truly interpreting the
input diagrams. To this end, we introduce MathVerse, an all-around visual math
benchmark designed for an equitable and in-depth evaluation of MLLMs. We
meticulously collect 2,612 high-quality, multi-subject math problems with
diagrams from publicly available sources. Each problem is then transformed by
human annotators into six distinct versions, each offering varying degrees of
information content in multi-modality, contributing to 15K test samples in
total. This approach allows MathVerse to comprehensively assess whether and how
much MLLMs can truly understand the visual diagrams for mathematical reasoning.
In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a
fine-grained assessment of the output answers. Rather than naively judging True
or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and
then score each step with detailed error analysis, which can reveal the
intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark
may provide unique insights to guide the future development of MLLMs. Project
page: https://mathverse-cuhk.github.io
MathVerse is a comprehensive visual math benchmark with CoT evaluation to assess MLLMs' true understanding and reasoning quality from visual diagrams.
Multi-modal Large Language Models (MLLMs) MathVerse Chain-of-Thought (CoT) GPT-4(V)