title: Do PhD-level LLMs Truly Grasp Elementary Addition? Probing Rule Learning
    vs. Memorization in Large Language Models
summary: Despite high benchmark scores, Large Language Models (LLMs) often fail simple
  problem, raising a critical question: Do LLMs learn mathematical principles or
  merely memorize patterns? Rather than designing increasingly complex benchmarks
  like recent works, we investigate this using elementary two-integer addition
  (0 to 2^{64}), probing two core properties: commutativity (A+B=B+A) and
  compositional generalization (via isomorphic symbolic mappings, e.g., 7
  rightarrow y). While state-of-the-art LLMs achieve 73.8-99.8\% accuracy on
  numerical addition, performance collapses to leq7.5\% under symbolic
  mapping, indicating failure to generalize learned rules. Non-monotonic
  performance scaling with digit count and frequent commutativity violations
  (over 1,700 cases of A+B neq B+A) further support this. Explicitly providing
  addition rules degrades performance by 81.2\% on average, while
  self-explanation maintains baseline accuracy, suggesting LLM arithmetic
  processing is misaligned with human-defined principles. Our findings indicate
  current LLMs rely on memory pattern over genuine rule learning, highlighting
  architectural limitations and the need for new approaches to achieve true
  mathematical reasoning.
paper_authors._id: 67fcc9a980568c7ef6180dcb
paper_authors.name: Yang Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-14T12:52:13.346Z
paper_authors._id: 62cd5917299c0c2e0e435847
paper_authors.avatarUrl: /avatars/b956b4feab86f6866c43cc87a44e25fc.svg
paper_authors.fullname: Yang Yan
paper_authors.type: user
paper_authors.user: kurileo
paper_authors._id: 67fcc9a980568c7ef6180dcc
paper_authors.name: Yu Lu
paper_authors._id: 67fcc9a980568c7ef6180dcd
paper_authors.name: Renjun Xu
paper_authors._id: 67fcc9a980568c7ef6180dce
paper_authors.name: Zhenzhong Lan
paper_title: Do PhD-level LLMs Truly Grasp Elementary Addition? Probing Rule Learning
    vs. Memorization in Large Language Models
paper_summary: Despite high benchmark scores, Large Language Models (LLMs) often fail simple
  problem, raising a critical question: Do LLMs learn mathematical principles or
  merely memorize patterns? Rather than designing increasingly complex benchmarks
  like recent works, we investigate this using elementary two-integer addition
  (0 to 2^{64}), probing two core properties: commutativity (A+B=B+A) and
  compositional generalization (via isomorphic symbolic mappings, e.g., 7
  rightarrow y). While state-of-the-art LLMs achieve 73.8-99.8\% accuracy on
  numerical addition, performance collapses to leq7.5\% under symbolic
  mapping, indicating failure to generalize learned rules. Non-monotonic
  performance scaling with digit count and frequent commutativity violations
  (over 1,700 cases of A+B neq B+A) further support this. Explicitly providing
  addition rules degrades performance by 81.2\% on average, while
  self-explanation maintains baseline accuracy, suggesting LLM arithmetic
  processing is misaligned with human-defined principles. Our findings indicate
  current LLMs rely on memory pattern over genuine rule learning, highlighting
  architectural limitations and the need for new approaches to achieve true
  mathematical reasoning.
paper_ai_summary: LLMs achieve high accuracy in numerical addition but fail to generalize to symbolic mappings and exhibit poor performance in commutativity, suggesting they rely on memorization rather than understanding mathematical principles.
paper_ai_keywords: Large Language Models elementary two-integer addition commutativity compositional generalization isomorphic symbolic mappings self-explanation