title: Mathematical Reasoning in Large Language Models: Assessing Logical and
    Arithmetic Errors across Wide Numerical Ranges
summary: Mathematical reasoning in Large Language Models (LLMs) is often evaluated
  using benchmarks with limited numerical ranges, failing to reflect real-world
  problem-solving across diverse scales. Furthermore, most existing evaluation
  methods only compare model outputs to ground-truth answers, obscuring insights
  into reasoning processes. To address these limitations, we introduce
  GSM-Ranges, a dataset generator derived from GSM8K that systematically perturbs
  numerical values in math problems to assess model robustness across varying
  numerical scales. Additionally, we propose a novel grading methodology that
  distinguishes between logical and non-logical errors, offering a more precise
  evaluation of reasoning processes beyond computational accuracy. Our
  experiments with various models reveal a significant increase in logical error
  rates-up to 14 percentage points-as numerical complexity rises, demonstrating a
  general weakness in reasoning with out-of-distribution numerical values.
  Moreover, while models demonstrate high accuracy on standalone arithmetic
  tasks, their performance deteriorates substantially when computations are
  embedded within word problems. These findings provide a comprehensive
  evaluation of LLMs' mathematical reasoning capabilities and inform future
  research directions for improving numerical generalization in language models.
paper_authors._id: 67af2b2d1f297f2bdacded89
paper_authors.name: Safal Shrestha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T12:04:44.693Z
paper_authors._id: 64cb922ec7f30fbf7b91a9a7
paper_authors.avatarUrl: /avatars/457eae5e56b9641ee5543146447d1755.svg
paper_authors.fullname: Safal Shrestha
paper_authors.type: user
paper_authors.user: safal312
paper_authors._id: 67af2b2d1f297f2bdacded8a
paper_authors.name: Minwu Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T12:00:37.837Z
paper_authors._id: 64e77b47d96966317b45eeb3
paper_authors.avatarUrl: /avatars/6b67eba3f15d6cd86ac3ad55c1daf166.svg
paper_authors.fullname: Minwu Kim
paper_authors.type: user
paper_authors.user: guactastesgood
paper_authors._id: 67af2b2d1f297f2bdacded8b
paper_authors.name: Keith Ross
paper_title: Mathematical Reasoning in Large Language Models: Assessing Logical and
    Arithmetic Errors across Wide Numerical Ranges
paper_summary: Mathematical reasoning in Large Language Models (LLMs) is often evaluated
  using benchmarks with limited numerical ranges, failing to reflect real-world
  problem-solving across diverse scales. Furthermore, most existing evaluation
  methods only compare model outputs to ground-truth answers, obscuring insights
  into reasoning processes. To address these limitations, we introduce
  GSM-Ranges, a dataset generator derived from GSM8K that systematically perturbs
  numerical values in math problems to assess model robustness across varying
  numerical scales. Additionally, we propose a novel grading methodology that
  distinguishes between logical and non-logical errors, offering a more precise
  evaluation of reasoning processes beyond computational accuracy. Our
  experiments with various models reveal a significant increase in logical error
  rates-up to 14 percentage points-as numerical complexity rises, demonstrating a
  general weakness in reasoning with out-of-distribution numerical values.
  Moreover, while models demonstrate high accuracy on standalone arithmetic
  tasks, their performance deteriorates substantially when computations are
  embedded within word problems. These findings provide a comprehensive
  evaluation of LLMs' mathematical reasoning capabilities and inform future
  research directions for improving numerical generalization in language models.
paper_ai_keywords: GSM-Ranges GSM8K numerical scales reasoning processes logical errors non-logical errors out-of-distribution numerical values standalone arithmetic tasks word problems numerical generalization