Is Your Model Really A Good Math Reasoner? Evaluating Mathematical
  Reasoning with Checklist
Exceptional mathematical reasoning ability is one of the key features that
demonstrate the power of large language models (LLMs). How to comprehensively
define and evaluate the mathematical abilities of LLMs, and even reflect the
user experience in real-world scenarios, has emerged as a critical issue.
Current benchmarks predominantly concentrate on problem-solving capabilities,
which presents a substantial risk of model overfitting and fails to accurately
represent genuine mathematical reasoning abilities. In this paper, we argue
that if a model really understands a problem, it should be robustly and readily
applied across a diverse array of tasks. Motivated by this, we introduce
MATHCHECK, a well-designed checklist for testing task generalization and
reasoning robustness, as well as an automatic tool to generate checklists
efficiently. MATHCHECK includes multiple mathematical reasoning tasks and
robustness test types to facilitate a comprehensive evaluation of both
mathematical reasoning ability and behavior testing. Utilizing MATHCHECK, we
develop MATHCHECK-GSM and MATHCHECK-GEO to assess mathematical textual
reasoning and multi-modal reasoning capabilities, respectively, serving as
upgraded versions of benchmarks including GSM8k, GeoQA, UniGeo, and Geometry3K.
We adopt MATHCHECK-GSM and MATHCHECK-GEO to evaluate over 20 LLMs and 11 MLLMs,
assessing their comprehensive mathematical reasoning abilities. Our results
demonstrate that while frontier LLMs like GPT-4o continue to excel in various
abilities on the checklist, many other model families exhibit a significant
decline. Further experiments indicate that, compared to traditional math
benchmarks, MATHCHECK better reflects true mathematical abilities and
represents mathematical intelligence more linearly, thereby supporting our
design. On our MATHCHECK, we can easily conduct detailed behavior analysis to
deeply investigate models.
6690e08eca8d295792a793c5
Zihao Zhou
claimed_verified
2024-07-15T09:00:52.571Z
65683c6efd8939c271447e9b
/avatars/e4a97ba3550ddcb24b906a7d4c87c861.svg
ZihaoZhou
user
ZihaoZhou
6690e08eca8d295792a793c6
Shudong Liu
admin_assigned
2024-07-12T09:43:18.299Z
654ce87af0b05673196a9f45
/avatars/7b9c854eb98e487e3057479b1c7860ac.svg
Shudong Liu
user
Sudanl
6690e08eca8d295792a793c7
Maizhen Ning
claimed_verified
2024-07-12T08:29:05.170Z
63b3fa84b7fec0adf64ed5df
/avatars/685009a89a6e843816376a2722650135.svg
Maizhen Ning
user
M-Ning
6690e08eca8d295792a793c8
Wei Liu
claimed_verified
2024-07-12T09:56:18.167Z
6458af46f4d212d780bd7c68
/avatars/832fd34bcc041b0b7b551873a459fc3c.svg
Wei Liu
user
PeterV09
6690e08eca8d295792a793c9
Jindong Wang
admin_assigned
2024-07-12T09:42:36.038Z
6204cc0d522e40b4a18d86e2
/avatars/18daf2de5671e711dc745388dd60569d.svg
Jindong Wang
user
jindongwang
6690e08eca8d295792a793ca
Derek F. Wong
claimed_verified
2025-03-20T10:55:27.561Z
648bd523805e4bcc541ec320
/avatars/443ca0c7cbeda3a08eb4af6a0e2da8bc.svg
Derek Wong
user
derekfw
6690e08eca8d295792a793cb
Xiaowei Huang
admin_assigned
2024-07-12T09:42:57.498Z
64a61a34ae2fee268a9272e7
/avatars/a15ff4eac84a17f592a2ed71f2e670c1.svg
Xiaowei Huang
user
kitaharatomoyo
6690e08eca8d295792a793cc
Qiufeng Wang
admin_assigned
2024-07-12T09:43:04.036Z
640f3984c025ddf618978589
/avatars/f8b11defb4a115f1af8d68b6d1228571.svg
Qiufeng Wang
user
bruceqfw
6690e08eca8d295792a793cd
Kaizhu Huang
Is Your Model Really A Good Math Reasoner? Evaluating Mathematical
  Reasoning with Checklist
Exceptional mathematical reasoning ability is one of the key features that
demonstrate the power of large language models (LLMs). How to comprehensively
define and evaluate the mathematical abilities of LLMs, and even reflect the
user experience in real-world scenarios, has emerged as a critical issue.
Current benchmarks predominantly concentrate on problem-solving capabilities,
which presents a substantial risk of model overfitting and fails to accurately
represent genuine mathematical reasoning abilities. In this paper, we argue
that if a model really understands a problem, it should be robustly and readily
applied across a diverse array of tasks. Motivated by this, we introduce
MATHCHECK, a well-designed checklist for testing task generalization and
reasoning robustness, as well as an automatic tool to generate checklists
efficiently. MATHCHECK includes multiple mathematical reasoning tasks and
robustness test types to facilitate a comprehensive evaluation of both
mathematical reasoning ability and behavior testing. Utilizing MATHCHECK, we
develop MATHCHECK-GSM and MATHCHECK-GEO to assess mathematical textual
reasoning and multi-modal reasoning capabilities, respectively, serving as
upgraded versions of benchmarks including GSM8k, GeoQA, UniGeo, and Geometry3K.
We adopt MATHCHECK-GSM and MATHCHECK-GEO to evaluate over 20 LLMs and 11 MLLMs,
assessing their comprehensive mathematical reasoning abilities. Our results
demonstrate that while frontier LLMs like GPT-4o continue to excel in various
abilities on the checklist, many other model families exhibit a significant
decline. Further experiments indicate that, compared to traditional math
benchmarks, MATHCHECK better reflects true mathematical abilities and
represents mathematical intelligence more linearly, thereby supporting our
design. On our MATHCHECK, we can easily conduct detailed behavior analysis to
deeply investigate models.
MATHCHECK introduces a comprehensive checklist and tool for evaluating mathematical reasoning and generalization of large language models across various tasks, improving on traditional benchmarks.
large language models LLMs mathematical reasoning MATHCHECK task generalization reasoning robustness MATHCHECK-GSM MATHCHECK-GEO GSM8k GeoQA UniGeo Geometry3K