title: B-STaR: Monitoring and Balancing Exploration and Exploitation in
    Self-Taught Reasoners
summary: In the absence of extensive human-annotated data for complex reasoning tasks,
  self-improvement -- where models are trained on their own outputs -- has
  emerged as a primary method for enhancing performance. However, the critical
  factors underlying the mechanism of these iterative self-improving methods
  remain poorly understood, such as under what conditions self-improvement is
  effective, and what are the bottlenecks in the current iterations. In this
  work, we identify and propose methods to monitor two pivotal factors in this
  iterative process: (1) the model's ability to generate sufficiently diverse
  responses (exploration); and (2) the effectiveness of external rewards in
  distinguishing high-quality candidates from lower-quality ones (exploitation).
  Using mathematical reasoning as a case study, we begin with a quantitative
  analysis to track the dynamics of exploration and exploitation, discovering
  that a model's exploratory capabilities rapidly deteriorate over iterations,
  and the effectiveness of exploiting external rewards diminishes as well.
  Motivated by these findings, we introduce B-STaR, a Self-Taught Reasoning
  framework that autonomously adjusts configurations across iterations to Balance
  exploration and exploitation, thereby optimizing the self-improving
  effectiveness based on the current policy model and available rewards. Our
  experiments on mathematical reasoning, coding, and commonsense reasoning
  demonstrate that B-STaR not only enhances the model's exploratory capabilities
  throughout training but also achieves a more effective balance between
  exploration and exploitation, leading to superior performance.
paper_authors._id: 676a23c19fc612bf4a3b93f6
paper_authors.name: Weihao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:49:25.881Z
paper_authors._id: 62751082b43ccfeef483424f
paper_authors.avatarUrl: /avatars/fec83e4478e7d1731ba6033328131852.svg
paper_authors.fullname: WeihaoZeng
paper_authors.type: user
paper_authors.user: AndrewZeng
paper_authors._id: 676a23c19fc612bf4a3b93f7
paper_authors.name: Yuzhen Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:05:42.772Z
paper_authors._id: 6462def82a83863b97c0611e
paper_authors.avatarUrl: /avatars/c03e9cc7d75b0266fcc56ecb6ee62148.svg
paper_authors.fullname: Yuzhen Huang
paper_authors.type: user
paper_authors.user: yuzhen17
paper_authors._id: 676a23c19fc612bf4a3b93f8
paper_authors.name: Lulu Zhao
paper_authors._id: 676a23c19fc612bf4a3b93f9
paper_authors.name: Yijun Wang
paper_authors._id: 676a23c19fc612bf4a3b93fa
paper_authors.name: Zifei Shan
paper_authors._id: 676a23c19fc612bf4a3b93fb
paper_authors.name: Junxian He
paper_title: B-STaR: Monitoring and Balancing Exploration and Exploitation in
    Self-Taught Reasoners
paper_summary: In the absence of extensive human-annotated data for complex reasoning tasks,
  self-improvement -- where models are trained on their own outputs -- has
  emerged as a primary method for enhancing performance. However, the critical
  factors underlying the mechanism of these iterative self-improving methods
  remain poorly understood, such as under what conditions self-improvement is
  effective, and what are the bottlenecks in the current iterations. In this
  work, we identify and propose methods to monitor two pivotal factors in this
  iterative process: (1) the model's ability to generate sufficiently diverse
  responses (exploration); and (2) the effectiveness of external rewards in
  distinguishing high-quality candidates from lower-quality ones (exploitation).
  Using mathematical reasoning as a case study, we begin with a quantitative
  analysis to track the dynamics of exploration and exploitation, discovering
  that a model's exploratory capabilities rapidly deteriorate over iterations,
  and the effectiveness of exploiting external rewards diminishes as well.
  Motivated by these findings, we introduce B-STaR, a Self-Taught Reasoning
  framework that autonomously adjusts configurations across iterations to Balance
  exploration and exploitation, thereby optimizing the self-improving
  effectiveness based on the current policy model and available rewards. Our
  experiments on mathematical reasoning, coding, and commonsense reasoning
  demonstrate that B-STaR not only enhances the model's exploratory capabilities
  throughout training but also achieves a more effective balance between
  exploration and exploitation, leading to superior performance.
paper_ai_summary: B-STaR framework autonomously adjusts configurations to enhance exploration and exploitation, improving performance in self-improving models.
paper_ai_keywords: self-improvement iterative self-improving methods exploration exploitation B-STaR mathematical reasoning coding commonsense reasoning