Mathematical Reasoning in Large Language Models: Assessing Logical and
  Arithmetic Errors across Wide Numerical Ranges
Mathematical reasoning in Large Language Models (LLMs) is often evaluated
using benchmarks with limited numerical ranges, failing to reflect real-world
problem-solving across diverse scales. Furthermore, most existing evaluation
methods only compare model outputs to ground-truth answers, obscuring insights
into reasoning processes. To address these limitations, we introduce
GSM-Ranges, a dataset generator derived from GSM8K that systematically perturbs
numerical values in math problems to assess model robustness across varying
numerical scales. Additionally, we propose a novel grading methodology that
distinguishes between logical and non-logical errors, offering a more precise
evaluation of reasoning processes beyond computational accuracy. Our
experiments with various models reveal a significant increase in logical error
rates-up to 14 percentage points-as numerical complexity rises, demonstrating a
general weakness in reasoning with out-of-distribution numerical values.
Moreover, while models demonstrate high accuracy on standalone arithmetic
tasks, their performance deteriorates substantially when computations are
embedded within word problems. These findings provide a comprehensive
evaluation of LLMs' mathematical reasoning capabilities and inform future
research directions for improving numerical generalization in language models.
67af2b2d1f297f2bdacded89
Safal Shrestha
claimed_verified
2025-02-14T12:04:44.693Z
64cb922ec7f30fbf7b91a9a7
/avatars/457eae5e56b9641ee5543146447d1755.svg
Safal Shrestha
user
safal312
67af2b2d1f297f2bdacded8a
Minwu Kim
claimed_verified
2025-02-14T12:00:37.837Z
64e77b47d96966317b45eeb3
/avatars/6b67eba3f15d6cd86ac3ad55c1daf166.svg
Minwu Kim
user
guactastesgood
67af2b2d1f297f2bdacded8b
Keith Ross
Mathematical Reasoning in Large Language Models: Assessing Logical and
  Arithmetic Errors across Wide Numerical Ranges
Mathematical reasoning in Large Language Models (LLMs) is often evaluated
using benchmarks with limited numerical ranges, failing to reflect real-world
problem-solving across diverse scales. Furthermore, most existing evaluation
methods only compare model outputs to ground-truth answers, obscuring insights
into reasoning processes. To address these limitations, we introduce
GSM-Ranges, a dataset generator derived from GSM8K that systematically perturbs
numerical values in math problems to assess model robustness across varying
numerical scales. Additionally, we propose a novel grading methodology that
distinguishes between logical and non-logical errors, offering a more precise
evaluation of reasoning processes beyond computational accuracy. Our
experiments with various models reveal a significant increase in logical error
rates-up to 14 percentage points-as numerical complexity rises, demonstrating a
general weakness in reasoning with out-of-distribution numerical values.
Moreover, while models demonstrate high accuracy on standalone arithmetic
tasks, their performance deteriorates substantially when computations are
embedded within word problems. These findings provide a comprehensive
evaluation of LLMs' mathematical reasoning capabilities and inform future
research directions for improving numerical generalization in language models.
GSM-Ranges evaluates large language models' mathematical reasoning across diverse numerical scales using perturbed values, revealing weaknesses in handling out-of-distribution numbers and word problems.
GSM-Ranges GSM8K numerical values robustness numerical scales grading methodology logical errors non-logical errors reasoning processes out-of-distribution numerical generalization word problems