B-STaR: Monitoring and Balancing Exploration and Exploitation in
  Self-Taught Reasoners
In the absence of extensive human-annotated data for complex reasoning tasks,
self-improvement -- where models are trained on their own outputs -- has
emerged as a primary method for enhancing performance. However, the critical
factors underlying the mechanism of these iterative self-improving methods
remain poorly understood, such as under what conditions self-improvement is
effective, and what are the bottlenecks in the current iterations. In this
work, we identify and propose methods to monitor two pivotal factors in this
iterative process: (1) the model's ability to generate sufficiently diverse
responses (exploration); and (2) the effectiveness of external rewards in
distinguishing high-quality candidates from lower-quality ones (exploitation).
Using mathematical reasoning as a case study, we begin with a quantitative
analysis to track the dynamics of exploration and exploitation, discovering
that a model's exploratory capabilities rapidly deteriorate over iterations,
and the effectiveness of exploiting external rewards diminishes as well.
Motivated by these findings, we introduce B-STaR, a Self-Taught Reasoning
framework that autonomously adjusts configurations across iterations to Balance
exploration and exploitation, thereby optimizing the self-improving
effectiveness based on the current policy model and available rewards. Our
experiments on mathematical reasoning, coding, and commonsense reasoning
demonstrate that B-STaR not only enhances the model's exploratory capabilities
throughout training but also achieves a more effective balance between
exploration and exploitation, leading to superior performance.
676a23c19fc612bf4a3b93f6
Weihao Zeng
claimed_verified
2025-01-15T08:49:25.881Z
62751082b43ccfeef483424f
/avatars/fec83e4478e7d1731ba6033328131852.svg
WeihaoZeng
user
AndrewZeng
676a23c19fc612bf4a3b93f7
Yuzhen Huang
claimed_verified
2025-01-03T14:05:42.772Z
6462def82a83863b97c0611e
/avatars/c03e9cc7d75b0266fcc56ecb6ee62148.svg
Yuzhen Huang
user
yuzhen17
676a23c19fc612bf4a3b93f8
Lulu Zhao
676a23c19fc612bf4a3b93f9
Yijun Wang
676a23c19fc612bf4a3b93fa
Zifei Shan
676a23c19fc612bf4a3b93fb
Junxian He
B-STaR: Monitoring and Balancing Exploration and Exploitation in
  Self-Taught Reasoners
In the absence of extensive human-annotated data for complex reasoning tasks,
self-improvement -- where models are trained on their own outputs -- has
emerged as a primary method for enhancing performance. However, the critical
factors underlying the mechanism of these iterative self-improving methods
remain poorly understood, such as under what conditions self-improvement is
effective, and what are the bottlenecks in the current iterations. In this
work, we identify and propose methods to monitor two pivotal factors in this
iterative process: (1) the model's ability to generate sufficiently diverse
responses (exploration); and (2) the effectiveness of external rewards in
distinguishing high-quality candidates from lower-quality ones (exploitation).
Using mathematical reasoning as a case study, we begin with a quantitative
analysis to track the dynamics of exploration and exploitation, discovering
that a model's exploratory capabilities rapidly deteriorate over iterations,
and the effectiveness of exploiting external rewards diminishes as well.
Motivated by these findings, we introduce B-STaR, a Self-Taught Reasoning
framework that autonomously adjusts configurations across iterations to Balance
exploration and exploitation, thereby optimizing the self-improving
effectiveness based on the current policy model and available rewards. Our
experiments on mathematical reasoning, coding, and commonsense reasoning
demonstrate that B-STaR not only enhances the model's exploratory capabilities
throughout training but also achieves a more effective balance between
exploration and exploitation, leading to superior performance.
B-STaR framework autonomously adjusts configurations to enhance exploration and exploitation, improving performance in self-improving models.
self-improvement iterative self-improving methods exploration exploitation B-STaR mathematical reasoning coding commonsense reasoning