To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic
  reasoning
Chain-of-thought (CoT) via prompting is the de facto method for eliciting
reasoning capabilities from large language models (LLMs). But for what kinds of
tasks is this extra ``thinking'' really helpful? To analyze this, we conducted
a quantitative meta-analysis covering over 100 papers using CoT and ran our own
evaluations of 20 datasets across 14 models. Our results show that CoT gives
strong performance benefits primarily on tasks involving math or logic, with
much smaller gains on other types of tasks. On MMLU, directly generating the
answer without CoT leads to almost identical accuracy as CoT unless the
question or model's response contains an equals sign, indicating symbolic
operations and reasoning. Following this finding, we analyze the behavior of
CoT on these problems by separating planning and execution and comparing
against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic
execution, but it underperforms relative to using a symbolic solver. Our
results indicate that CoT can be applied selectively, maintaining performance
while saving inference costs. Furthermore, they suggest a need to move beyond
prompt-based CoT to new paradigms that better leverage intermediate computation
across the whole range of LLM applications.
66eb8d13e0f57407e06a5fd7
Zayne Sprague
extracted_pending
2024-09-19T02:31:49.154Z
64e78a03e3953cd90bcad620
https://cdn-avatars.huggingface.co/v1/production/uploads/64e78a03e3953cd90bcad620/Rj_-xLJUsxdRmNhvRbssq.jpeg
Zayne Sprague
user
Zaynes
66eb8d13e0f57407e06a5fd8
Fangcong Yin
admin_assigned
2024-09-19T08:28:01.263Z
64efa8748602335a044cd97f
/avatars/0ab5df922cb0ce4abe7aed35e7b9100c.svg
Fangcong Yin
user
fcyin
66eb8d13e0f57407e06a5fd9
Juan Diego Rodriguez
admin_assigned
2024-09-19T08:28:36.069Z
64baabce565b827f7e661372
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/t4F7d1hyEGHzNfy7JMYgs.png
Juan Diego Rodriguez
user
juand-r
66eb8d13e0f57407e06a5fda
Dongwei Jiang
admin_assigned
2024-09-19T08:28:45.446Z
63eba2774dcaaf087638e3d6
https://cdn-avatars.huggingface.co/v1/production/uploads/1676386919413-noauth.jpeg
Jiang
user
Dongwei
66eb8d13e0f57407e06a5fdb
Manya Wadhwa
admin_assigned
2024-09-19T08:29:03.051Z
655ab2ccc11dee7f7e6db119
/avatars/1c13e338cd4cc0b4eb681ed8f33abf19.svg
Manya Wadhwa
user
wadhma
66eb8d13e0f57407e06a5fdc
Prasann Singhal
admin_assigned
2024-09-19T08:29:08.894Z
613cb3e1c7a43c281cd417a2
/avatars/69123ba49c2aa1cd9f3cc5746f4839dc.svg
Prasann Singhal
user
PrasannSinghal
66eb8d13e0f57407e06a5fdd
Xinyu Zhao
claimed_verified
2024-09-25T07:34:06.074Z
65f356f3244d1b4066bcc62b
/avatars/9eedf36a95e6d573bd3f8c0f0d16a9d6.svg
Xinyu (Lucy) Zhao
user
xyz-zy
66eb8d13e0f57407e06a5fde
Xi Ye
66eb8d13e0f57407e06a5fdf
Kyle Mahowald
admin_assigned
2024-09-19T08:29:59.830Z
6669940e3ed628e8f4c3128d
/avatars/399dcd69d99b4e1fdaf6ef9fdf39224e.svg
Kyle Mahowald
user
mahowak
66eb8d13e0f57407e06a5fe0
Greg Durrett
admin_assigned
2024-09-19T08:30:05.707Z
65be9918b54ab5b37d1b67a7
/avatars/9953707affb6881724c8efb2abf0c668.svg
Greg Durrett
user
gregdurrett
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic
  reasoning
Chain-of-thought (CoT) via prompting is the de facto method for eliciting
reasoning capabilities from large language models (LLMs). But for what kinds of
tasks is this extra ``thinking'' really helpful? To analyze this, we conducted
a quantitative meta-analysis covering over 100 papers using CoT and ran our own
evaluations of 20 datasets across 14 models. Our results show that CoT gives
strong performance benefits primarily on tasks involving math or logic, with
much smaller gains on other types of tasks. On MMLU, directly generating the
answer without CoT leads to almost identical accuracy as CoT unless the
question or model's response contains an equals sign, indicating symbolic
operations and reasoning. Following this finding, we analyze the behavior of
CoT on these problems by separating planning and execution and comparing
against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic
execution, but it underperforms relative to using a symbolic solver. Our
results indicate that CoT can be applied selectively, maintaining performance
while saving inference costs. Furthermore, they suggest a need to move beyond
prompt-based CoT to new paradigms that better leverage intermediate computation
across the whole range of LLM applications.
Chain-of-thought prompting enhances large language model performance primarily in math or logic tasks by improving symbolic reasoning, but it is less beneficial in other domains and can be selectively applied to save inference costs.
chain-of-thought large language models MMLU symbolic operations reasoning prompt-based symbolic execution tool-augmented LLMs