title: To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic
    reasoning
summary: Chain-of-thought (CoT) via prompting is the de facto method for eliciting
  reasoning capabilities from large language models (LLMs). But for what kinds of
  tasks is this extra ``thinking'' really helpful? To analyze this, we conducted
  a quantitative meta-analysis covering over 100 papers using CoT and ran our own
  evaluations of 20 datasets across 14 models. Our results show that CoT gives
  strong performance benefits primarily on tasks involving math or logic, with
  much smaller gains on other types of tasks. On MMLU, directly generating the
  answer without CoT leads to almost identical accuracy as CoT unless the
  question or model's response contains an equals sign, indicating symbolic
  operations and reasoning. Following this finding, we analyze the behavior of
  CoT on these problems by separating planning and execution and comparing
  against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic
  execution, but it underperforms relative to using a symbolic solver. Our
  results indicate that CoT can be applied selectively, maintaining performance
  while saving inference costs. Furthermore, they suggest a need to move beyond
  prompt-based CoT to new paradigms that better leverage intermediate computation
  across the whole range of LLM applications.
paper_authors._id: 66eb8d13e0f57407e06a5fd7
paper_authors.name: Zayne Sprague
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-09-19T02:31:49.154Z
paper_authors._id: 64e78a03e3953cd90bcad620
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e78a03e3953cd90bcad620/Rj_-xLJUsxdRmNhvRbssq.jpeg
paper_authors.fullname: Zayne Sprague
paper_authors.type: user
paper_authors.user: Zaynes
paper_authors._id: 66eb8d13e0f57407e06a5fd8
paper_authors.name: Fangcong Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:28:01.263Z
paper_authors._id: 64efa8748602335a044cd97f
paper_authors.avatarUrl: /avatars/0ab5df922cb0ce4abe7aed35e7b9100c.svg
paper_authors.fullname: Fangcong Yin
paper_authors.type: user
paper_authors.user: fcyin
paper_authors._id: 66eb8d13e0f57407e06a5fd9
paper_authors.name: Juan Diego Rodriguez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:28:36.069Z
paper_authors._id: 64baabce565b827f7e661372
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/t4F7d1hyEGHzNfy7JMYgs.png
paper_authors.fullname: Juan Diego Rodriguez
paper_authors.type: user
paper_authors.user: juand-r
paper_authors._id: 66eb8d13e0f57407e06a5fda
paper_authors.name: Dongwei Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:28:45.446Z
paper_authors._id: 63eba2774dcaaf087638e3d6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676386919413-noauth.jpeg
paper_authors.fullname: Jiang
paper_authors.type: user
paper_authors.user: Dongwei
paper_authors._id: 66eb8d13e0f57407e06a5fdb
paper_authors.name: Manya Wadhwa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:29:03.051Z
paper_authors._id: 655ab2ccc11dee7f7e6db119
paper_authors.avatarUrl: /avatars/1c13e338cd4cc0b4eb681ed8f33abf19.svg
paper_authors.fullname: Manya Wadhwa
paper_authors.type: user
paper_authors.user: wadhma
paper_authors._id: 66eb8d13e0f57407e06a5fdc
paper_authors.name: Prasann Singhal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:29:08.894Z
paper_authors._id: 613cb3e1c7a43c281cd417a2
paper_authors.avatarUrl: /avatars/69123ba49c2aa1cd9f3cc5746f4839dc.svg
paper_authors.fullname: Prasann Singhal
paper_authors.type: user
paper_authors.user: PrasannSinghal
paper_authors._id: 66eb8d13e0f57407e06a5fdd
paper_authors.name: Xinyu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:34:06.074Z
paper_authors._id: 65f356f3244d1b4066bcc62b
paper_authors.avatarUrl: /avatars/9eedf36a95e6d573bd3f8c0f0d16a9d6.svg
paper_authors.fullname: Xinyu (Lucy) Zhao
paper_authors.type: user
paper_authors.user: xyz-zy
paper_authors._id: 66eb8d13e0f57407e06a5fde
paper_authors.name: Xi Ye
paper_authors._id: 66eb8d13e0f57407e06a5fdf
paper_authors.name: Kyle Mahowald
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:29:59.830Z
paper_authors._id: 6669940e3ed628e8f4c3128d
paper_authors.avatarUrl: /avatars/399dcd69d99b4e1fdaf6ef9fdf39224e.svg
paper_authors.fullname: Kyle Mahowald
paper_authors.type: user
paper_authors.user: mahowak
paper_authors._id: 66eb8d13e0f57407e06a5fe0
paper_authors.name: Greg Durrett
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:30:05.707Z
paper_authors._id: 65be9918b54ab5b37d1b67a7
paper_authors.avatarUrl: /avatars/8237eb89aa1f9847e7f55c11925a108a.svg
paper_authors.fullname: Greg Durrett
paper_authors.type: user
paper_authors.user: gregdurrett
paper_title: To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic
    reasoning
paper_summary: Chain-of-thought (CoT) via prompting is the de facto method for eliciting
  reasoning capabilities from large language models (LLMs). But for what kinds of
  tasks is this extra ``thinking'' really helpful? To analyze this, we conducted
  a quantitative meta-analysis covering over 100 papers using CoT and ran our own
  evaluations of 20 datasets across 14 models. Our results show that CoT gives
  strong performance benefits primarily on tasks involving math or logic, with
  much smaller gains on other types of tasks. On MMLU, directly generating the
  answer without CoT leads to almost identical accuracy as CoT unless the
  question or model's response contains an equals sign, indicating symbolic
  operations and reasoning. Following this finding, we analyze the behavior of
  CoT on these problems by separating planning and execution and comparing
  against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic
  execution, but it underperforms relative to using a symbolic solver. Our
  results indicate that CoT can be applied selectively, maintaining performance
  while saving inference costs. Furthermore, they suggest a need to move beyond
  prompt-based CoT to new paradigms that better leverage intermediate computation
  across the whole range of LLM applications.
paper_ai_keywords: chain-of-thought large language models (LLMs) quantitative meta-analysis MMLU (Multiple-Choice Machine Learning Understanding) symbolic operations symbolic reasoning symbolic execution tool-augmented LLMs intermediate computation