title: One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual
    Reasoning in Mathematical LLMs
summary: Leveraging mathematical Large Language Models (LLMs) for proof generation is
  a fundamental topic in LLMs research. We argue that the ability of current LLMs
  to prove statements largely depends on whether they have encountered the
  relevant proof process during training. This reliance limits their deeper
  understanding of mathematical theorems and related concepts. Inspired by the
  pedagogical method of "proof by counterexamples" commonly used in human
  mathematics education, our work aims to enhance LLMs' ability to conduct
  mathematical reasoning and proof through counterexamples. Specifically, we
  manually create a high-quality, university-level mathematical benchmark,
  CounterMATH, which requires LLMs to prove mathematical statements by providing
  counterexamples, thereby assessing their grasp of mathematical concepts.
  Additionally, we develop a data engineering framework to automatically obtain
  training data for further model improvement. Extensive experiments and detailed
  analyses demonstrate that CounterMATH is challenging, indicating that LLMs,
  such as OpenAI o1, have insufficient counterexample-driven proof capabilities.
  Moreover, our exploration into model training reveals that strengthening LLMs'
  counterexample-driven conceptual reasoning abilities is crucial for improving
  their overall mathematical capabilities. We believe that our work offers new
  perspectives on the community of mathematical LLMs.
paper_authors._id: 67b40e56bffd44cc85976ecd
paper_authors.name: Yinghui Li
paper_authors._id: 67b40e56bffd44cc85976ece
paper_authors.name: Jiayi Kuang
paper_authors._id: 67b40e56bffd44cc85976ecf
paper_authors.name: Haojing Huang
paper_authors._id: 67b40e56bffd44cc85976ed0
paper_authors.name: Zhikun Xu
paper_authors._id: 67b40e56bffd44cc85976ed1
paper_authors.name: Xinnian Liang
paper_authors._id: 67b40e56bffd44cc85976ed2
paper_authors.name: Yi Yu
paper_authors._id: 67b40e56bffd44cc85976ed3
paper_authors.name: Wenlian Lu
paper_authors._id: 67b40e56bffd44cc85976ed4
paper_authors.name: Yangning Li
paper_authors._id: 67b40e56bffd44cc85976ed5
paper_authors.name: Xiaoyu Tan
paper_authors._id: 67b40e56bffd44cc85976ed6
paper_authors.name: Chao Qu
paper_authors._id: 67b40e56bffd44cc85976ed7
paper_authors.name: Ying Shen
paper_authors._id: 67b40e56bffd44cc85976ed8
paper_authors.name: Hai-Tao Zheng
paper_authors._id: 67b40e56bffd44cc85976ed9
paper_authors.name: Philip S. Yu
paper_title: One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual
    Reasoning in Mathematical LLMs
paper_summary: Leveraging mathematical Large Language Models (LLMs) for proof generation is
  a fundamental topic in LLMs research. We argue that the ability of current LLMs
  to prove statements largely depends on whether they have encountered the
  relevant proof process during training. This reliance limits their deeper
  understanding of mathematical theorems and related concepts. Inspired by the
  pedagogical method of "proof by counterexamples" commonly used in human
  mathematics education, our work aims to enhance LLMs' ability to conduct
  mathematical reasoning and proof through counterexamples. Specifically, we
  manually create a high-quality, university-level mathematical benchmark,
  CounterMATH, which requires LLMs to prove mathematical statements by providing
  counterexamples, thereby assessing their grasp of mathematical concepts.
  Additionally, we develop a data engineering framework to automatically obtain
  training data for further model improvement. Extensive experiments and detailed
  analyses demonstrate that CounterMATH is challenging, indicating that LLMs,
  such as OpenAI o1, have insufficient counterexample-driven proof capabilities.
  Moreover, our exploration into model training reveals that strengthening LLMs'
  counterexample-driven conceptual reasoning abilities is crucial for improving
  their overall mathematical capabilities. We believe that our work offers new
  perspectives on the community of mathematical LLMs.
paper_ai_keywords: proof generation Large Language Models (LLMs) proof process mathematical theorems proof by counterexamples CounterMATH mathematical reasoning counterexample-driven proof training data mathematical capabilities conceptual reasoning