title: Is Your Model Really A Good Math Reasoner? Evaluating Mathematical
    Reasoning with Checklist
summary: Exceptional mathematical reasoning ability is one of the key features that
  demonstrate the power of large language models (LLMs). How to comprehensively
  define and evaluate the mathematical abilities of LLMs, and even reflect the
  user experience in real-world scenarios, has emerged as a critical issue.
  Current benchmarks predominantly concentrate on problem-solving capabilities,
  which presents a substantial risk of model overfitting and fails to accurately
  represent genuine mathematical reasoning abilities. In this paper, we argue
  that if a model really understands a problem, it should be robustly and readily
  applied across a diverse array of tasks. Motivated by this, we introduce
  MATHCHECK, a well-designed checklist for testing task generalization and
  reasoning robustness, as well as an automatic tool to generate checklists
  efficiently. MATHCHECK includes multiple mathematical reasoning tasks and
  robustness test types to facilitate a comprehensive evaluation of both
  mathematical reasoning ability and behavior testing. Utilizing MATHCHECK, we
  develop MATHCHECK-GSM and MATHCHECK-GEO to assess mathematical textual
  reasoning and multi-modal reasoning capabilities, respectively, serving as
  upgraded versions of benchmarks including GSM8k, GeoQA, UniGeo, and Geometry3K.
  We adopt MATHCHECK-GSM and MATHCHECK-GEO to evaluate over 20 LLMs and 11 MLLMs,
  assessing their comprehensive mathematical reasoning abilities. Our results
  demonstrate that while frontier LLMs like GPT-4o continue to excel in various
  abilities on the checklist, many other model families exhibit a significant
  decline. Further experiments indicate that, compared to traditional math
  benchmarks, MATHCHECK better reflects true mathematical abilities and
  represents mathematical intelligence more linearly, thereby supporting our
  design. On our MATHCHECK, we can easily conduct detailed behavior analysis to
  deeply investigate models.
paper_authors._id: 6690e08eca8d295792a793c5
paper_authors.name: Zihao Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T09:00:52.571Z
paper_authors._id: 65683c6efd8939c271447e9b
paper_authors.avatarUrl: /avatars/e4a97ba3550ddcb24b906a7d4c87c861.svg
paper_authors.fullname: ZihaoZhou
paper_authors.type: user
paper_authors.user: ZihaoZhou
paper_authors._id: 6690e08eca8d295792a793c6
paper_authors.name: Shudong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T09:43:18.299Z
paper_authors._id: 654ce87af0b05673196a9f45
paper_authors.avatarUrl: /avatars/7b9c854eb98e487e3057479b1c7860ac.svg
paper_authors.fullname: Shudong Liu
paper_authors.type: user
paper_authors.user: Sudanl
paper_authors._id: 6690e08eca8d295792a793c7
paper_authors.name: Maizhen Ning
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T08:29:05.170Z
paper_authors._id: 63b3fa84b7fec0adf64ed5df
paper_authors.avatarUrl: /avatars/685009a89a6e843816376a2722650135.svg
paper_authors.fullname: Maizhen Ning
paper_authors.type: user
paper_authors.user: M-Ning
paper_authors._id: 6690e08eca8d295792a793c8
paper_authors.name: Wei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T09:56:18.167Z
paper_authors._id: 6458af46f4d212d780bd7c68
paper_authors.avatarUrl: /avatars/832fd34bcc041b0b7b551873a459fc3c.svg
paper_authors.fullname: Wei Liu
paper_authors.type: user
paper_authors.user: PeterV09
paper_authors._id: 6690e08eca8d295792a793c9
paper_authors.name: Jindong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T09:42:36.038Z
paper_authors._id: 6204cc0d522e40b4a18d86e2
paper_authors.avatarUrl: /avatars/18daf2de5671e711dc745388dd60569d.svg
paper_authors.fullname: Jindong Wang
paper_authors.type: user
paper_authors.user: jindongwang
paper_authors._id: 6690e08eca8d295792a793ca
paper_authors.name: Derek F. Wong
paper_authors._id: 6690e08eca8d295792a793cb
paper_authors.name: Xiaowei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T09:42:57.498Z
paper_authors._id: 64a61a34ae2fee268a9272e7
paper_authors.avatarUrl: /avatars/a15ff4eac84a17f592a2ed71f2e670c1.svg
paper_authors.fullname: Xiaowei Huang
paper_authors.type: user
paper_authors.user: kitaharatomoyo
paper_authors._id: 6690e08eca8d295792a793cc
paper_authors.name: Qiufeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T09:43:04.036Z
paper_authors._id: 640f3984c025ddf618978589
paper_authors.avatarUrl: /avatars/f8b11defb4a115f1af8d68b6d1228571.svg
paper_authors.fullname: Qiufeng Wang
paper_authors.type: user
paper_authors.user: bruceqfw
paper_authors._id: 6690e08eca8d295792a793cd
paper_authors.name: Kaizhu Huang
paper_title: Is Your Model Really A Good Math Reasoner? Evaluating Mathematical
    Reasoning with Checklist
paper_summary: Exceptional mathematical reasoning ability is one of the key features that
  demonstrate the power of large language models (LLMs). How to comprehensively
  define and evaluate the mathematical abilities of LLMs, and even reflect the
  user experience in real-world scenarios, has emerged as a critical issue.
  Current benchmarks predominantly concentrate on problem-solving capabilities,
  which presents a substantial risk of model overfitting and fails to accurately
  represent genuine mathematical reasoning abilities. In this paper, we argue
  that if a model really understands a problem, it should be robustly and readily
  applied across a diverse array of tasks. Motivated by this, we introduce
  MATHCHECK, a well-designed checklist for testing task generalization and
  reasoning robustness, as well as an automatic tool to generate checklists
  efficiently. MATHCHECK includes multiple mathematical reasoning tasks and
  robustness test types to facilitate a comprehensive evaluation of both
  mathematical reasoning ability and behavior testing. Utilizing MATHCHECK, we
  develop MATHCHECK-GSM and MATHCHECK-GEO to assess mathematical textual
  reasoning and multi-modal reasoning capabilities, respectively, serving as
  upgraded versions of benchmarks including GSM8k, GeoQA, UniGeo, and Geometry3K.
  We adopt MATHCHECK-GSM and MATHCHECK-GEO to evaluate over 20 LLMs and 11 MLLMs,
  assessing their comprehensive mathematical reasoning abilities. Our results
  demonstrate that while frontier LLMs like GPT-4o continue to excel in various
  abilities on the checklist, many other model families exhibit a significant
  decline. Further experiments indicate that, compared to traditional math
  benchmarks, MATHCHECK better reflects true mathematical abilities and
  represents mathematical intelligence more linearly, thereby supporting our
  design. On our MATHCHECK, we can easily conduct detailed behavior analysis to
  deeply investigate models.
paper_ai_keywords: MATHCHECK MATHCHECK-GSM MATHCHECK-GEO mathematical reasoning tasks robustness test types behavior testing mathematical textual reasoning multi-modal reasoning capabilities GSM8k GeoQA UniGeo Geometry3K LLMs MLLMs GPT-4o true mathematical abilities mathematical intelligence