title: Demystifying Long Chain-of-Thought Reasoning in LLMs
summary: Scaling inference compute enhances reasoning in large language models (LLMs),
  with long chains-of-thought (CoTs) enabling strategies like backtracking and
  error correction. Reinforcement learning (RL) has emerged as a crucial method
  for developing these capabilities, yet the conditions under which long CoTs
  emerge remain unclear, and RL training requires careful design choices. In this
  study, we systematically investigate the mechanics of long CoT reasoning,
  identifying the key factors that enable models to generate long CoT
  trajectories. Through extensive supervised fine-tuning (SFT) and RL
  experiments, we present four main findings: (1) While SFT is not strictly
  necessary, it simplifies training and improves efficiency; (2) Reasoning
  capabilities tend to emerge with increased training compute, but their
  development is not guaranteed, making reward shaping crucial for stabilizing
  CoT length growth; (3) Scaling verifiable reward signals is critical for RL. We
  find that leveraging noisy, web-extracted solutions with filtering mechanisms
  shows strong potential, particularly for out-of-distribution (OOD) tasks such
  as STEM reasoning; and (4) Core abilities like error correction are inherently
  present in base models, but incentivizing these skills effectively for complex
  tasks via RL demands significant compute, and measuring their emergence
  requires a nuanced approach. These insights provide practical guidance for
  optimizing training strategies to enhance long CoT reasoning in LLMs. Our code
  is available at: https://github.com/eddycmu/demystify-long-cot.
paper_authors._id: 67a42c079a4fb11b11cc4f6f
paper_authors.name: Edward Yeo
paper_authors._id: 67a42c079a4fb11b11cc4f70
paper_authors.name: Yuxuan Tong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:34:50.960Z
paper_authors._id: 6448e1fbe988635a3d6aa97d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eG4R9-3hgrimttP7ep3dN.jpeg
paper_authors.fullname: Shawn/Yuxuan Tong
paper_authors.type: user
paper_authors.user: tongyx361
paper_authors._id: 67a42c079a4fb11b11cc4f71
paper_authors.name: Morry Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:34:57.424Z
paper_authors._id: 65bb14f139c4e7087640a91c
paper_authors.avatarUrl: /avatars/dbf75dd161d22b4511e9fccff6afc515.svg
paper_authors.fullname: Morry Niu
paper_authors.type: user
paper_authors.user: bl1ndbot
paper_authors._id: 67a42c079a4fb11b11cc4f72
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:35:04.994Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_authors._id: 67a42c079a4fb11b11cc4f73
paper_authors.name: Xiang Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:35:19.222Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_title: Demystifying Long Chain-of-Thought Reasoning in LLMs
paper_summary: Scaling inference compute enhances reasoning in large language models (LLMs),
  with long chains-of-thought (CoTs) enabling strategies like backtracking and
  error correction. Reinforcement learning (RL) has emerged as a crucial method
  for developing these capabilities, yet the conditions under which long CoTs
  emerge remain unclear, and RL training requires careful design choices. In this
  study, we systematically investigate the mechanics of long CoT reasoning,
  identifying the key factors that enable models to generate long CoT
  trajectories. Through extensive supervised fine-tuning (SFT) and RL
  experiments, we present four main findings: (1) While SFT is not strictly
  necessary, it simplifies training and improves efficiency; (2) Reasoning
  capabilities tend to emerge with increased training compute, but their
  development is not guaranteed, making reward shaping crucial for stabilizing
  CoT length growth; (3) Scaling verifiable reward signals is critical for RL. We
  find that leveraging noisy, web-extracted solutions with filtering mechanisms
  shows strong potential, particularly for out-of-distribution (OOD) tasks such
  as STEM reasoning; and (4) Core abilities like error correction are inherently
  present in base models, but incentivizing these skills effectively for complex
  tasks via RL demands significant compute, and measuring their emergence
  requires a nuanced approach. These insights provide practical guidance for
  optimizing training strategies to enhance long CoT reasoning in LLMs. Our code
  is available at: https://github.com/eddycmu/demystify-long-cot.
paper_ai_summary: Investigation into long chains-of-thought reasoning in large language models reveals the critical role of training compute, reward shaping, and verifiable reward signals in enabling and measuring this capability.
paper_ai_keywords: large language models long chains-of-thought backtracking error correction reinforcement learning supervised fine-tuning training compute reward shaping verifiable reward signals web-extracted solutions filtering mechanisms out-of-distribution tasks STEM reasoning core abilities