Demystifying Long Chain-of-Thought Reasoning in LLMs
Scaling inference compute enhances reasoning in large language models (LLMs),
with long chains-of-thought (CoTs) enabling strategies like backtracking and
error correction. Reinforcement learning (RL) has emerged as a crucial method
for developing these capabilities, yet the conditions under which long CoTs
emerge remain unclear, and RL training requires careful design choices. In this
study, we systematically investigate the mechanics of long CoT reasoning,
identifying the key factors that enable models to generate long CoT
trajectories. Through extensive supervised fine-tuning (SFT) and RL
experiments, we present four main findings: (1) While SFT is not strictly
necessary, it simplifies training and improves efficiency; (2) Reasoning
capabilities tend to emerge with increased training compute, but their
development is not guaranteed, making reward shaping crucial for stabilizing
CoT length growth; (3) Scaling verifiable reward signals is critical for RL. We
find that leveraging noisy, web-extracted solutions with filtering mechanisms
shows strong potential, particularly for out-of-distribution (OOD) tasks such
as STEM reasoning; and (4) Core abilities like error correction are inherently
present in base models, but incentivizing these skills effectively for complex
tasks via RL demands significant compute, and measuring their emergence
requires a nuanced approach. These insights provide practical guidance for
optimizing training strategies to enhance long CoT reasoning in LLMs. Our code
is available at: https://github.com/eddycmu/demystify-long-cot.
67a42c079a4fb11b11cc4f6f
Edward Yeo
67a42c079a4fb11b11cc4f70
Yuxuan Tong
admin_assigned
2025-02-06T14:34:50.960Z
6448e1fbe988635a3d6aa97d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eG4R9-3hgrimttP7ep3dN.jpeg
Shawn/Yuxuan Tong
user
tongyx361
67a42c079a4fb11b11cc4f71
Morry Niu
admin_assigned
2025-02-06T14:34:57.424Z
65bb14f139c4e7087640a91c
/avatars/dbf75dd161d22b4511e9fccff6afc515.svg
Morry Niu
user
bl1ndbot
67a42c079a4fb11b11cc4f72
Graham Neubig
admin_assigned
2025-02-06T14:35:04.994Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
67a42c079a4fb11b11cc4f73
Xiang Yue
admin_assigned
2025-02-06T14:35:19.222Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
Demystifying Long Chain-of-Thought Reasoning in LLMs
Scaling inference compute enhances reasoning in large language models (LLMs),
with long chains-of-thought (CoTs) enabling strategies like backtracking and
error correction. Reinforcement learning (RL) has emerged as a crucial method
for developing these capabilities, yet the conditions under which long CoTs
emerge remain unclear, and RL training requires careful design choices. In this
study, we systematically investigate the mechanics of long CoT reasoning,
identifying the key factors that enable models to generate long CoT
trajectories. Through extensive supervised fine-tuning (SFT) and RL
experiments, we present four main findings: (1) While SFT is not strictly
necessary, it simplifies training and improves efficiency; (2) Reasoning
capabilities tend to emerge with increased training compute, but their
development is not guaranteed, making reward shaping crucial for stabilizing
CoT length growth; (3) Scaling verifiable reward signals is critical for RL. We
find that leveraging noisy, web-extracted solutions with filtering mechanisms
shows strong potential, particularly for out-of-distribution (OOD) tasks such
as STEM reasoning; and (4) Core abilities like error correction are inherently
present in base models, but incentivizing these skills effectively for complex
tasks via RL demands significant compute, and measuring their emergence
requires a nuanced approach. These insights provide practical guidance for
optimizing training strategies to enhance long CoT reasoning in LLMs. Our code
is available at: https://github.com/eddycmu/demystify-long-cot.
Investigation into long chains-of-thought reasoning in large language models reveals the critical role of training compute, reward shaping, and verifiable reward signals in enabling and measuring this capability.
large language models long chains-of-thought backtracking error correction reinforcement learning supervised fine-tuning training compute reward shaping verifiable reward signals web-extracted solutions filtering mechanisms out-of-distribution tasks STEM reasoning core abilities