title: FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language
    Models
summary: Formal mathematical reasoning remains a critical challenge for artificial
  intelligence, hindered by limitations of existing benchmarks in scope and
  scale. To address this, we present FormalMATH, a large-scale Lean4 benchmark
  comprising 5,560 formally verified problems spanning from high-school Olympiad
  challenges to undergraduate-level theorems across diverse domains (e.g.,
  algebra, applied mathematics, calculus, number theory, and discrete
  mathematics). To mitigate the inefficiency of manual formalization, we
  introduce a novel human-in-the-loop autoformalization pipeline that integrates:
  (1) specialized large language models (LLMs) for statement autoformalization,
  (2) multi-LLM semantic verification, and (3) negation-based disproof filtering
  strategies using off-the-shelf LLM-based provers. This approach reduces expert
  annotation costs by retaining 72.09% of statements before manual verification
  while ensuring fidelity to the original natural-language problems. Our
  evaluation of state-of-the-art LLM-based theorem provers reveals significant
  limitations: even the strongest models achieve only 16.46% success rate under
  practical sampling budgets, exhibiting pronounced domain bias (e.g., excelling
  in algebra but failing in calculus) and over-reliance on simplified automation
  tactics. Notably, we identify a counterintuitive inverse relationship between
  natural-language solution guidance and proof success in chain-of-thought
  reasoning scenarios, suggesting that human-written informal reasoning
  introduces noise rather than clarity in the formal reasoning settings. We
  believe that FormalMATH provides a robust benchmark for benchmarking formal
  mathematical reasoning.
paper_authors._id: 6819742e0d1c56fe9124fe3a
paper_authors.name: Zhouliang Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:34:10.190Z
paper_authors._id: 62a80fe3ac97233f1625235a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a80fe3ac97233f1625235a/_rGtpqdY7OEBz3pyqb6fE.jpeg
paper_authors.fullname: Zhouliang Yu
paper_authors.type: user
paper_authors.user: zhouliang
paper_authors._id: 6819742e0d1c56fe9124fe3b
paper_authors.name: Ruotian Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:48:20.491Z
paper_authors._id: 662f2c8435ab6df959b005de
paper_authors.avatarUrl: /avatars/3e30053ecbe9cc14b5e1eb2b014755de.svg
paper_authors.fullname: ruotian peng
paper_authors.type: user
paper_authors.user: prt66
paper_authors._id: 6819742e0d1c56fe9124fe3c
paper_authors.name: Keyi Ding
paper_authors._id: 6819742e0d1c56fe9124fe3d
paper_authors.name: Yizhe Li
paper_authors._id: 6819742e0d1c56fe9124fe3e
paper_authors.name: Zhongyuan Peng
paper_authors._id: 6819742e0d1c56fe9124fe3f
paper_authors.name: Minghao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:33:31.975Z
paper_authors._id: 6417d9ea8f689506e7148417
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
paper_authors.fullname: minghao
paper_authors.type: user
paper_authors.user: Liam-Liu
paper_authors._id: 6819742e0d1c56fe9124fe40
paper_authors.name: Yifan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T14:10:20.040Z
paper_authors._id: 647bf082aba7062fe5c51ca9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/p4lY9IjHiWZETKmFq1mtU.jpeg
paper_authors.fullname: Yifan Zhang
paper_authors.type: user
paper_authors.user: yifAI
paper_authors._id: 6819742e0d1c56fe9124fe41
paper_authors.name: Zheng Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:49:20.735Z
paper_authors._id: 649da6b4599302cdb9bc232b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/DxQT6LCDTZvyGUUe2t19c.jpeg
paper_authors.fullname: Zheng Yuan
paper_authors.type: user
paper_authors.user: ZhengYuan
paper_authors._id: 6819742e0d1c56fe9124fe42
paper_authors.name: Huajian Xin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:49:28.104Z
paper_authors._id: 6532a060a78e70d19c669103
paper_authors.avatarUrl: /avatars/3cc9309b0e31da0fb83f1c3ef87dbe9f.svg
paper_authors.fullname: HuajianXin
paper_authors.type: user
paper_authors.user: HuajianXin
paper_authors._id: 6819742e0d1c56fe9124fe43
paper_authors.name: Wenhao Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:49:44.482Z
paper_authors._id: 641e5bf65f274a0a92c2f6a2
paper_authors.avatarUrl: /avatars/c15a54c51998c0e6367685e8e1737ec9.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: EZ-hwh
paper_authors._id: 6819742e0d1c56fe9124fe44
paper_authors.name: Yandong Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:49:51.642Z
paper_authors._id: 643c21735fcffe09fb68a46f
paper_authors.avatarUrl: /avatars/76aabacd318aa954d4c53094ad456056.svg
paper_authors.fullname: Yandong Wen
paper_authors.type: user
paper_authors.user: ydwen
paper_authors._id: 6819742e0d1c56fe9124fe45
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:49:59.764Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 6819742e0d1c56fe9124fe46
paper_authors.name: Weiyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:50:07.063Z
paper_authors._id: 648905d1a15c43c791d4381f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648905d1a15c43c791d4381f/GpqGBzsLiMHX0gWZEz3qn.jpeg
paper_authors.fullname: Weiyang Liu
paper_authors.type: user
paper_authors.user: wy1iu
paper_title: FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language
    Models
paper_summary: Formal mathematical reasoning remains a critical challenge for artificial
  intelligence, hindered by limitations of existing benchmarks in scope and
  scale. To address this, we present FormalMATH, a large-scale Lean4 benchmark
  comprising 5,560 formally verified problems spanning from high-school Olympiad
  challenges to undergraduate-level theorems across diverse domains (e.g.,
  algebra, applied mathematics, calculus, number theory, and discrete
  mathematics). To mitigate the inefficiency of manual formalization, we
  introduce a novel human-in-the-loop autoformalization pipeline that integrates:
  (1) specialized large language models (LLMs) for statement autoformalization,
  (2) multi-LLM semantic verification, and (3) negation-based disproof filtering
  strategies using off-the-shelf LLM-based provers. This approach reduces expert
  annotation costs by retaining 72.09% of statements before manual verification
  while ensuring fidelity to the original natural-language problems. Our
  evaluation of state-of-the-art LLM-based theorem provers reveals significant
  limitations: even the strongest models achieve only 16.46% success rate under
  practical sampling budgets, exhibiting pronounced domain bias (e.g., excelling
  in algebra but failing in calculus) and over-reliance on simplified automation
  tactics. Notably, we identify a counterintuitive inverse relationship between
  natural-language solution guidance and proof success in chain-of-thought
  reasoning scenarios, suggesting that human-written informal reasoning
  introduces noise rather than clarity in the formal reasoning settings. We
  believe that FormalMATH provides a robust benchmark for benchmarking formal
  mathematical reasoning.
paper_ai_summary: FormalMATH, a large-scale Lean4 benchmark, introduces an autoformalization pipeline to reduce manual annotation costs and identifies limitations of existing LLM-based theorem provers in formal mathematical reasoning.
paper_ai_keywords: Lean4 autoformalization large language models multi-LLM semantic verification negation-based disproof filtering formal mathematical reasoning theorem provers domain bias chain-of-thought reasoning