title: How Much Backtracking is Enough? Exploring the Interplay of SFT and RL
    in Enhancing LLM Reasoning
summary: Recent breakthroughs in large language models (LLMs) have effectively
  improved their reasoning abilities, particularly on mathematical and logical
  problems that have verifiable answers, through techniques such as supervised
  finetuning (SFT) and reinforcement learning (RL). Prior research indicates that
  RL effectively internalizes search strategies, enabling long chain-of-thought
  (CoT) reasoning, with backtracking emerging naturally as a learned capability.
  However, the precise benefits of backtracking, specifically, how significantly
  it contributes to reasoning improvements and the optimal extent of its use,
  remain poorly understood. In this work, we systematically investigate the
  dynamics between SFT and RL on eight reasoning tasks: Countdown, Sudoku, Arc
  1D, Geometry, Color Cube Rotation, List Functions, Zebra Puzzles, and Self
  Reference. Our findings highlight that short CoT sequences used in SFT as a
  warm-up do have moderate contribution to RL training, compared with cold-start
  RL; however such contribution diminishes when tasks become increasingly
  difficult. Motivated by this observation, we construct synthetic datasets
  varying systematically in the number of backtracking steps and conduct
  controlled experiments to isolate the influence of either the correctness
  (content) or the structure (i.e., backtrack frequency). We find that (1) longer
  CoT with backtracks generally induce better and more stable RL training, (2)
  more challenging problems with larger search space tend to need higher numbers
  of backtracks during the SFT stage. Additionally, we demonstrate through
  experiments on distilled data that RL training is largely unaffected by the
  correctness of long CoT sequences, suggesting that RL prioritizes structural
  patterns over content correctness. Collectively, our results offer practical
  insights into designing optimal training strategies to effectively scale
  reasoning in LLMs.
paper_authors._id: 683f21ed6ba11d78e3e383f6
paper_authors.name: Hongyi James Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:01:32.571Z
paper_authors._id: 65f7c56fc6356b5cc5ab8245
paper_authors.avatarUrl: /avatars/ff8d3f3526e915f269274cc0dc5ca8ef.svg
paper_authors.fullname: James Cai
paper_authors.type: user
paper_authors.user: jamescai20
paper_authors._id: 683f21ed6ba11d78e3e383f7
paper_authors.name: Junlin Wang
paper_authors._id: 683f21ed6ba11d78e3e383f8
paper_authors.name: Xiaoyin Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-06-03T16:25:17.851Z
paper_authors._id: 65d66cb2b06abf924b07ff76
paper_authors.avatarUrl: /avatars/de94e2fe07040b7dc3053bcaafa64ffe.svg
paper_authors.fullname: Xiaoyin Chen
paper_authors.type: user
paper_authors.user: chenyn66
paper_authors._id: 683f21ed6ba11d78e3e383f9
paper_authors.name: Bhuwan Dhingra
paper_title: How Much Backtracking is Enough? Exploring the Interplay of SFT and RL
    in Enhancing LLM Reasoning
paper_summary: Recent breakthroughs in large language models (LLMs) have effectively
  improved their reasoning abilities, particularly on mathematical and logical
  problems that have verifiable answers, through techniques such as supervised
  finetuning (SFT) and reinforcement learning (RL). Prior research indicates that
  RL effectively internalizes search strategies, enabling long chain-of-thought
  (CoT) reasoning, with backtracking emerging naturally as a learned capability.
  However, the precise benefits of backtracking, specifically, how significantly
  it contributes to reasoning improvements and the optimal extent of its use,
  remain poorly understood. In this work, we systematically investigate the
  dynamics between SFT and RL on eight reasoning tasks: Countdown, Sudoku, Arc
  1D, Geometry, Color Cube Rotation, List Functions, Zebra Puzzles, and Self
  Reference. Our findings highlight that short CoT sequences used in SFT as a
  warm-up do have moderate contribution to RL training, compared with cold-start
  RL; however such contribution diminishes when tasks become increasingly
  difficult. Motivated by this observation, we construct synthetic datasets
  varying systematically in the number of backtracking steps and conduct
  controlled experiments to isolate the influence of either the correctness
  (content) or the structure (i.e., backtrack frequency). We find that (1) longer
  CoT with backtracks generally induce better and more stable RL training, (2)
  more challenging problems with larger search space tend to need higher numbers
  of backtracks during the SFT stage. Additionally, we demonstrate through
  experiments on distilled data that RL training is largely unaffected by the
  correctness of long CoT sequences, suggesting that RL prioritizes structural
  patterns over content correctness. Collectively, our results offer practical
  insights into designing optimal training strategies to effectively scale
  reasoning in LLMs.
paper_ai_summary: This study investigates the interplay between supervised fine-tuning and reinforcement learning in large language models, focusing on the role of backtracking in enhancing reasoning capabilities across various tasks.
paper_ai_keywords: supervised finetuning reinforcement learning chain-of-thought backtracking countdown sudoku arc 1d geometry color cube rotation list functions zebra puzzles self reference synthetic datasets distilled data