title: Language Models are Symbolic Learners in Arithmetic
summary: Large Language Models (LLMs) are thought to struggle with arithmetic learning
  due to the inherent differences between language modeling and numerical
  computation, but concrete evidence has been lacking. This work responds to this
  claim through a two-side experiment. We first investigate whether LLMs leverage
  partial products during arithmetic learning. We find that although LLMs can
  identify some partial products after learning, they fail to leverage them for
  arithmetic tasks, conversely. We then explore how LLMs approach arithmetic
  symbolically by breaking tasks into subgroups, hypothesizing that difficulties
  arise from subgroup complexity and selection. Our results show that when
  subgroup complexity is fixed, LLMs treat a collection of different arithmetic
  operations similarly. By analyzing position-level accuracy across different
  training sizes, we further observe that it follows a U-shaped pattern: LLMs
  quickly learn the easiest patterns at the first and last positions, while
  progressively learning the more difficult patterns in the middle positions.
  This suggests that LLMs select subgroup following an easy-to-hard paradigm
  during learning. Our work confirms that LLMs are pure symbolic learners in
  arithmetic tasks and underscores the importance of understanding them deeply
  through subgroup-level quantification.
paper_authors._id: 671b23c017de97fe524d5828
paper_authors.name: Chunyuan Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:54:46.917Z
paper_authors._id: 634b9914dcf125e4da02498b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634b9914dcf125e4da02498b/crRgFroWq5U6XWtvlTXSZ.jpeg
paper_authors.fullname: Chunyuan Deng
paper_authors.type: user
paper_authors.user: CharlesDDDD
paper_authors._id: 671b23c017de97fe524d5829
paper_authors.name: Zhiqi Li
paper_authors._id: 671b23c017de97fe524d582a
paper_authors.name: Roy Xie
paper_authors._id: 671b23c017de97fe524d582b
paper_authors.name: Ruidi Chang
paper_authors._id: 671b23c017de97fe524d582c
paper_authors.name: Hanjie Chen
paper_title: Language Models are Symbolic Learners in Arithmetic
paper_summary: Large Language Models (LLMs) are thought to struggle with arithmetic learning
  due to the inherent differences between language modeling and numerical
  computation, but concrete evidence has been lacking. This work responds to this
  claim through a two-side experiment. We first investigate whether LLMs leverage
  partial products during arithmetic learning. We find that although LLMs can
  identify some partial products after learning, they fail to leverage them for
  arithmetic tasks, conversely. We then explore how LLMs approach arithmetic
  symbolically by breaking tasks into subgroups, hypothesizing that difficulties
  arise from subgroup complexity and selection. Our results show that when
  subgroup complexity is fixed, LLMs treat a collection of different arithmetic
  operations similarly. By analyzing position-level accuracy across different
  training sizes, we further observe that it follows a U-shaped pattern: LLMs
  quickly learn the easiest patterns at the first and last positions, while
  progressively learning the more difficult patterns in the middle positions.
  This suggests that LLMs select subgroup following an easy-to-hard paradigm
  during learning. Our work confirms that LLMs are pure symbolic learners in
  arithmetic tasks and underscores the importance of understanding them deeply
  through subgroup-level quantification.
paper_ai_summary: LLMs struggle with arithmetic learning despite being able to identify partial products, treating different operations similarly when subgroup complexity is fixed, and learning patterns in an easy-to-hard manner.
paper_ai_keywords: Large Language Models LLMs arithmetic learning partial products symbolic learners subgroup complexity U-shaped pattern position-level accuracy