title: Not All LLM Reasoners Are Created Equal
summary: We study the depth of grade-school math (GSM) problem-solving capabilities of
  LLMs. To this end, we evaluate their performance on pairs of existing math word
  problems together so that the answer to the second problem depends on correctly
  answering the first problem. Our findings reveal a significant reasoning gap in
  most LLMs, that is performance difference between solving the compositional
  pairs and solving each question independently. This gap is more pronounced in
  smaller, more cost-efficient, and math-specialized models. Moreover,
  instruction-tuning recipes and code generation have varying effects across LLM
  sizes, while finetuning on GSM can lead to task overfitting. Our analysis
  indicates that large reasoning gaps are not because of test-set leakage, but
  due to distraction from additional context and poor second-hop reasoning.
  Overall, LLMs exhibit systematic differences in their reasoning abilities,
  despite what their performance on standard benchmarks indicates.
paper_authors._id: 66fe4d83b7c3b3c0f298f78b
paper_authors.name: Arian Hosseini
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-03T07:53:39.980Z
paper_authors._id: 62e447e9640206c5d21e5ff6
paper_authors.avatarUrl: /avatars/42d8247c2ffa98a4c6255ce84051f1b9.svg
paper_authors.fullname: Arian Hosseini
paper_authors.type: user
paper_authors.user: arianhosseini
paper_authors._id: 66fe4d83b7c3b3c0f298f78c
paper_authors.name: Alessandro Sordoni
paper_authors._id: 66fe4d83b7c3b3c0f298f78d
paper_authors.name: Daniel Toyama
paper_authors._id: 66fe4d83b7c3b3c0f298f78e
paper_authors.name: Aaron Courville
paper_authors._id: 66fe4d83b7c3b3c0f298f78f
paper_authors.name: Rishabh Agarwal
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-03T07:53:39.980Z
paper_authors._id: 647875e37b370854241b89a8
paper_authors.avatarUrl: /avatars/fd25f7e5960e4e132b9ee689f114820b.svg
paper_authors.fullname: Rishabh Agarwal
paper_authors.type: user
paper_authors.user: agarwl
paper_title: Not All LLM Reasoners Are Created Equal
paper_summary: We study the depth of grade-school math (GSM) problem-solving capabilities of
  LLMs. To this end, we evaluate their performance on pairs of existing math word
  problems together so that the answer to the second problem depends on correctly
  answering the first problem. Our findings reveal a significant reasoning gap in
  most LLMs, that is performance difference between solving the compositional
  pairs and solving each question independently. This gap is more pronounced in
  smaller, more cost-efficient, and math-specialized models. Moreover,
  instruction-tuning recipes and code generation have varying effects across LLM
  sizes, while finetuning on GSM can lead to task overfitting. Our analysis
  indicates that large reasoning gaps are not because of test-set leakage, but
  due to distraction from additional context and poor second-hop reasoning.
  Overall, LLMs exhibit systematic differences in their reasoning abilities,
  despite what their performance on standard benchmarks indicates.
paper_ai_summary: LLMs display a significant reasoning gap when solving dependent math problems, which is more pronounced in smaller and math-specialized models, and is attributed to distraction from additional context and poor second-hop reasoning.