title: We-Math: Does Your Large Multimodal Model Achieve Human-like
    Mathematical Reasoning?
summary: Visual mathematical reasoning, as a fundamental visual reasoning ability, has
  received widespread attention from the Large Multimodal Models (LMMs)
  community. Existing benchmarks, such as MathVista and MathVerse, focus more on
  the result-oriented performance but neglect the underlying principles in
  knowledge acquisition and generalization. Inspired by human-like mathematical
  reasoning, we introduce WE-MATH, the first benchmark specifically designed to
  explore the problem-solving principles beyond end-to-end performance. We
  meticulously collect and categorize 6.5K visual math problems, spanning 67
  hierarchical knowledge concepts and five layers of knowledge granularity. We
  decompose composite problems into sub-problems according to the required
  knowledge concepts and introduce a novel four-dimensional metric, namely
  Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery
  (CM), and Rote Memorization (RM), to hierarchically assess inherent issues in
  LMMs' reasoning process. With WE-MATH, we conduct a thorough evaluation of
  existing LMMs in visual mathematical reasoning and reveal a negative
  correlation between solving steps and problem-specific performance. We confirm
  the IK issue of LMMs can be effectively improved via knowledge augmentation
  strategies. More notably, the primary challenge of GPT-4o has significantly
  transitioned from IK to IG, establishing it as the first LMM advancing towards
  the knowledge generalization stage. In contrast, other LMMs exhibit a marked
  inclination towards Rote Memorization - they correctly solve composite problems
  involving multiple knowledge concepts yet fail to answer sub-problems. We
  anticipate that WE-MATH will open new pathways for advancements in visual
  mathematical reasoning for LMMs. The WE-MATH data and evaluation code are
  available at https://github.com/We-Math/We-Math.
paper_authors._id: 66835909bad5fded13ebcdcd
paper_authors.name: Runqi Qiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:52:11.000Z
paper_authors._id: 6683a05e74fb1736a4b7c934
paper_authors.avatarUrl: /avatars/35292253cf1882080be353c4b8c9ca3b.svg
paper_authors.fullname: QRQ
paper_authors.type: user
paper_authors.user: RichardQRQ
paper_authors._id: 66835909bad5fded13ebcdce
paper_authors.name: Qiuna Tan
paper_authors._id: 66835909bad5fded13ebcdcf
paper_authors.name: Guanting Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-02T08:00:53.082Z
paper_authors._id: 61cd4b833dd34ba1985e0753
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
paper_authors.fullname: KABI
paper_authors.type: user
paper_authors.user: dongguanting
paper_authors._id: 66835909bad5fded13ebcdd0
paper_authors.name: Minhui Wu
paper_authors._id: 66835909bad5fded13ebcdd1
paper_authors.name: Chong Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-02T08:01:24.607Z
paper_authors._id: 657e77a7efc0f84bcb7ff6ce
paper_authors.avatarUrl: /avatars/384e0d8e1ffdf99451ab40e9fcba93ec.svg
paper_authors.fullname: Chong Sun
paper_authors.type: user
paper_authors.user: Csun1992
paper_authors._id: 66835909bad5fded13ebcdd2
paper_authors.name: Xiaoshuai Song
paper_authors._id: 66835909bad5fded13ebcdd3
paper_authors.name: Zhuoma GongQue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:34:25.993Z
paper_authors._id: 6762b973686e574cab4717ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/icacS5LkbX2S46b3Dqbni.png
paper_authors.fullname: zhuoma
paper_authors.type: user
paper_authors.user: Ethereal1l
paper_authors._id: 66835909bad5fded13ebcdd4
paper_authors.name: Shanglin Lei
paper_authors._id: 66835909bad5fded13ebcdd5
paper_authors.name: Zhe Wei
paper_authors._id: 66835909bad5fded13ebcdd6
paper_authors.name: Miaoxuan Zhang
paper_authors._id: 66835909bad5fded13ebcdd7
paper_authors.name: Runfeng Qiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-02T08:20:20.931Z
paper_authors._id: 668362a5f8fd8217cc20b493
paper_authors.avatarUrl: /avatars/bc40a0aafdcbf22dc018ad8652a272fe.svg
paper_authors.fullname: Runfeng Qiao
paper_authors.type: user
paper_authors.user: Arsoeir
paper_authors._id: 66835909bad5fded13ebcdd8
paper_authors.name: Yifan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T14:34:59.169Z
paper_authors._id: 66615dd1c8bbe034b4b82029
paper_authors.avatarUrl: /avatars/7ef1ddf2fa356efbeb2b3aa12284a1f0.svg
paper_authors.fullname: Zhang YiFan
paper_authors.type: user
paper_authors.user: zhangyifan666
paper_authors._id: 66835909bad5fded13ebcdd9
paper_authors.name: Xiao Zong
paper_authors._id: 66835909bad5fded13ebcdda
paper_authors.name: Yida Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-02T08:19:19.230Z
paper_authors._id: 662b19df4f711ee4e1f5bf9b
paper_authors.avatarUrl: /avatars/d2a0eab67f71d0eab28dffa7214186eb.svg
paper_authors.fullname: Yida Xu
paper_authors.type: user
paper_authors.user: yidada
paper_authors._id: 66835909bad5fded13ebcddb
paper_authors.name: Muxi Diao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:40:00.199Z
paper_authors._id: 6768c97367e4b4606a3c9cec
paper_authors.avatarUrl: /avatars/5ddafe7a05828366f66c79072556f370.svg
paper_authors.fullname: diaomuxi
paper_authors.type: user
paper_authors.user: diaomuxi
paper_authors._id: 66835909bad5fded13ebcddc
paper_authors.name: Zhimin Bao
paper_authors._id: 66835909bad5fded13ebcddd
paper_authors.name: Chen Li
paper_authors._id: 66835909bad5fded13ebcdde
paper_authors.name: Honggang Zhang
paper_title: We-Math: Does Your Large Multimodal Model Achieve Human-like
    Mathematical Reasoning?
paper_summary: Visual mathematical reasoning, as a fundamental visual reasoning ability, has
  received widespread attention from the Large Multimodal Models (LMMs)
  community. Existing benchmarks, such as MathVista and MathVerse, focus more on
  the result-oriented performance but neglect the underlying principles in
  knowledge acquisition and generalization. Inspired by human-like mathematical
  reasoning, we introduce WE-MATH, the first benchmark specifically designed to
  explore the problem-solving principles beyond end-to-end performance. We
  meticulously collect and categorize 6.5K visual math problems, spanning 67
  hierarchical knowledge concepts and five layers of knowledge granularity. We
  decompose composite problems into sub-problems according to the required
  knowledge concepts and introduce a novel four-dimensional metric, namely
  Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery
  (CM), and Rote Memorization (RM), to hierarchically assess inherent issues in
  LMMs' reasoning process. With WE-MATH, we conduct a thorough evaluation of
  existing LMMs in visual mathematical reasoning and reveal a negative
  correlation between solving steps and problem-specific performance. We confirm
  the IK issue of LMMs can be effectively improved via knowledge augmentation
  strategies. More notably, the primary challenge of GPT-4o has significantly
  transitioned from IK to IG, establishing it as the first LMM advancing towards
  the knowledge generalization stage. In contrast, other LMMs exhibit a marked
  inclination towards Rote Memorization - they correctly solve composite problems
  involving multiple knowledge concepts yet fail to answer sub-problems. We
  anticipate that WE-MATH will open new pathways for advancements in visual
  mathematical reasoning for LMMs. The WE-MATH data and evaluation code are
  available at https://github.com/We-Math/We-Math.
paper_ai_keywords: Large Multimodal Models (LMMs) MathVista MathVerse WE-MATH visual math problems composite problems sub-problems knowledge concepts knowledge granularity Insufficient Knowledge (IK) Inadequate Generalization (IG) Focus Mastery (CM) Rote Memorization (RM) hierarchical assessment GPT-4o knowledge augmentation visual mathematical reasoning