title: Large Language Models and Mathematical Reasoning Failures
summary: This paper investigates the mathematical reasoning capabilities of large
  language models (LLMs) using 50 newly constructed high-school-level word
  problems. Unlike prior studies that focus solely on answer correctness, we
  rigorously analyze both final answers and solution steps to identify reasoning
  failures. Evaluating eight state-of-the-art models - including Mixtral, Llama,
  Gemini, GPT-4o, and OpenAI's o1 variants - we find that while newer models
  (e.g., o3-mini, deepseek-r1) achieve higher accuracy, all models exhibit errors
  in spatial reasoning, strategic planning, and arithmetic, sometimes producing
  correct answers through flawed logic. Common failure modes include unwarranted
  assumptions, over-reliance on numerical patterns, and difficulty translating
  physical intuition into mathematical steps. Manual analysis reveals that models
  struggle with problems requiring multi-step deduction or real-world knowledge,
  despite possessing broad mathematical knowledge. Our results underscore the
  importance of evaluating reasoning processes, not just answers, and caution
  against overestimating LLMs' problem-solving proficiency. The study highlights
  persistent gaps in LLMs' generalization abilities, emphasizing the need for
  targeted improvements in structured reasoning and constraint handling.
paper_authors._id: 67b435c29e5685b308a8edac
paper_authors.name: Johan Boye
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-18T07:24:50.956Z
paper_authors._id: 65bcbc01d6d0ffbceb8b2e6e
paper_authors.avatarUrl: /avatars/73edb2d6b7b11208439ac88b365079e8.svg
paper_authors.fullname: Johan Boye
paper_authors.type: user
paper_authors.user: jboye
paper_authors._id: 67b435c29e5685b308a8edad
paper_authors.name: Birger Moell
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:30:49.328Z
paper_authors._id: 6033e34a9aa44495c80dd043
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1614079701740-6033e34a9aa44495c80dd043.jpeg
paper_authors.fullname: Birger Moell
paper_authors.type: user
paper_authors.user: birgermoell
paper_title: Large Language Models and Mathematical Reasoning Failures
paper_summary: This paper investigates the mathematical reasoning capabilities of large
  language models (LLMs) using 50 newly constructed high-school-level word
  problems. Unlike prior studies that focus solely on answer correctness, we
  rigorously analyze both final answers and solution steps to identify reasoning
  failures. Evaluating eight state-of-the-art models - including Mixtral, Llama,
  Gemini, GPT-4o, and OpenAI's o1 variants - we find that while newer models
  (e.g., o3-mini, deepseek-r1) achieve higher accuracy, all models exhibit errors
  in spatial reasoning, strategic planning, and arithmetic, sometimes producing
  correct answers through flawed logic. Common failure modes include unwarranted
  assumptions, over-reliance on numerical patterns, and difficulty translating
  physical intuition into mathematical steps. Manual analysis reveals that models
  struggle with problems requiring multi-step deduction or real-world knowledge,
  despite possessing broad mathematical knowledge. Our results underscore the
  importance of evaluating reasoning processes, not just answers, and caution
  against overestimating LLMs' problem-solving proficiency. The study highlights
  persistent gaps in LLMs' generalization abilities, emphasizing the need for
  targeted improvements in structured reasoning and constraint handling.
paper_ai_summary: Evaluation of large language models on high-school-level math problems shows that while newer models improve accuracy, they still fail in spatial reasoning, strategic planning, arithmetic, and multi-step deduction.
paper_ai_keywords: mathematical reasoning large language models word problems Mixtral Llama Gemini GPT-4 spatial reasoning strategic planning arithmetic solution steps reasoning failures manual analysis structured reasoning constraint handling