title: Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four
    Habits of Highly Effective STaRs
summary: Test-time inference has emerged as a powerful paradigm for enabling language
  models to ``think'' longer and more carefully about complex challenges, much
  like skilled human experts. While reinforcement learning (RL) can drive
  self-improvement in language models on verifiable tasks, some models exhibit
  substantial gains while others quickly plateau. For instance, we find that
  Qwen-2.5-3B far exceeds Llama-3.2-3B under identical RL training for the game
  of Countdown. This discrepancy raises a critical question: what intrinsic
  properties enable effective self-improvement? We introduce a framework to
  investigate this question by analyzing four key cognitive behaviors --
  verification, backtracking, subgoal setting, and backward chaining -- that both
  expert human problem solvers and successful language models employ. Our study
  reveals that Qwen naturally exhibits these reasoning behaviors, whereas Llama
  initially lacks them. In systematic experimentation with controlled behavioral
  datasets, we find that priming Llama with examples containing these reasoning
  behaviors enables substantial improvements during RL, matching or exceeding
  Qwen's performance. Importantly, the presence of reasoning behaviors, rather
  than correctness of answers, proves to be the critical factor -- models primed
  with incorrect solutions containing proper reasoning patterns achieve
  comparable performance to those trained on correct solutions. Finally,
  leveraging continued pretraining with OpenWebMath data, filtered to amplify
  reasoning behaviors, enables the Llama model to match Qwen's self-improvement
  trajectory. Our findings establish a fundamental relationship between initial
  reasoning behaviors and the capacity for improvement, explaining why some
  language models effectively utilize additional computation while others
  plateau.
paper_authors._id: 67c68adc0457c9f809c22df8
paper_authors.name: Kanishk Gandhi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:35:01.161Z
paper_authors._id: 63e6a880f2e9a8f22c5a1630
paper_authors.avatarUrl: /avatars/53b57690fe052ce6882bbfc87b11567c.svg
paper_authors.fullname: Kanishk Gandhi
paper_authors.type: user
paper_authors.user: obiwan96
paper_authors._id: 67c68adc0457c9f809c22df9
paper_authors.name: Ayush Chakravarthy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:04:44.344Z
paper_authors._id: 624f9e3d07bd004fb855f5e9
paper_authors.avatarUrl: /avatars/86a349cd4053bc0317e27e75a51c69fa.svg
paper_authors.fullname: Ayush Chakravarthy
paper_authors.type: user
paper_authors.user: ayushchakravarthy
paper_authors._id: 67c68adc0457c9f809c22dfa
paper_authors.name: Anikait Singh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:05:05.759Z
paper_authors._id: 6511ee845b7e52b0251fdee9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6511ee845b7e52b0251fdee9/hTIwiIYBGOVnIrxtpri83.png
paper_authors.fullname: Anikait Singh
paper_authors.type: user
paper_authors.user: Asap7772
paper_authors._id: 67c68adc0457c9f809c22dfb
paper_authors.name: Nathan Lile
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:34:58.582Z
paper_authors._id: 61aa15fd8a9625ebfe284286
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61aa15fd8a9625ebfe284286/KaGzIeijcgcN15JErCqft.jpeg
paper_authors.fullname: nathan lile
paper_authors.type: user
paper_authors.user: nlile
paper_authors._id: 67c68adc0457c9f809c22dfc
paper_authors.name: Noah D. Goodman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:05:12.186Z
paper_authors._id: 67321274c1f20c742bcf7a8d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ltcQhre6eDRVzn6Vbbyhu.png
paper_authors.fullname: Noah D. Goodman
paper_authors.type: user
paper_authors.user: ngoodman
paper_title: Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four
    Habits of Highly Effective STaRs
paper_summary: Test-time inference has emerged as a powerful paradigm for enabling language
  models to ``think'' longer and more carefully about complex challenges, much
  like skilled human experts. While reinforcement learning (RL) can drive
  self-improvement in language models on verifiable tasks, some models exhibit
  substantial gains while others quickly plateau. For instance, we find that
  Qwen-2.5-3B far exceeds Llama-3.2-3B under identical RL training for the game
  of Countdown. This discrepancy raises a critical question: what intrinsic
  properties enable effective self-improvement? We introduce a framework to
  investigate this question by analyzing four key cognitive behaviors --
  verification, backtracking, subgoal setting, and backward chaining -- that both
  expert human problem solvers and successful language models employ. Our study
  reveals that Qwen naturally exhibits these reasoning behaviors, whereas Llama
  initially lacks them. In systematic experimentation with controlled behavioral
  datasets, we find that priming Llama with examples containing these reasoning
  behaviors enables substantial improvements during RL, matching or exceeding
  Qwen's performance. Importantly, the presence of reasoning behaviors, rather
  than correctness of answers, proves to be the critical factor -- models primed
  with incorrect solutions containing proper reasoning patterns achieve
  comparable performance to those trained on correct solutions. Finally,
  leveraging continued pretraining with OpenWebMath data, filtered to amplify
  reasoning behaviors, enables the Llama model to match Qwen's self-improvement
  trajectory. Our findings establish a fundamental relationship between initial
  reasoning behaviors and the capacity for improvement, explaining why some
  language models effectively utilize additional computation while others
  plateau.
paper_ai_keywords: reinforcement learning (RL) self-improvement sequence language models Qwen-2.5-3B Llama-3.2-3B Countdown intrinsic traits keylocated behaviors verification backtracking sub questions backward finding reasoning behaviors controlled datasets priming OpenWebMath continued pretraining