Thought-Augmented Policy Optimization: Bridging External Guidance and
  Internal Capabilities
Reinforcement learning (RL) has emerged as an effective method for training
reasoning models. However, existing RL approaches typically bias the model's
output distribution toward reward-maximizing paths without introducing external
knowledge. This limits their exploration capacity and results in a narrower
reasoning capability boundary compared to base models. To address this
limitation, we propose TAPO (Thought-Augmented Policy Optimization), a novel
framework that augments RL by incorporating external high-level guidance
("thought patterns"). By adaptively integrating structured thoughts during
training, TAPO effectively balances model-internal exploration and external
guidance exploitation. Extensive experiments show that our approach
significantly outperforms GRPO by 99% on AIME, 41% on AMC, and 17% on Minerva
Math. Notably, these high-level thought patterns, abstracted from only 500
prior samples, generalize effectively across various tasks and models. This
highlights TAPO's potential for broader applications across multiple tasks and
domains. Our further analysis reveals that introducing external guidance
produces powerful reasoning models with superior explainability of inference
behavior and enhanced output readability.
68306ffdff038ca6400a153a
Jinyang Wu
claimed_verified
2025-05-26T08:13:57.397Z
6747de57f8cab58c22ec94a2
/avatars/5bae0341862fac24564781c0fa32aac5.svg
Jinyang Wu
user
Jinyang23
68306ffdff038ca6400a153b
Chonghua Liao
claimed_verified
2025-05-26T08:16:00.056Z
667fdaee20ee9ac417c7708c
/avatars/69dfba6ff392643af1dcfe8af0a42ae9.svg
Chonghua Liao
user
ChonghuaLiao
68306ffdff038ca6400a153c
Mingkuan Feng
claimed_verified
2025-06-16T09:52:52.674Z
6708e132e0c87dc0e55fc6bb
/avatars/7ba826abc56641b9418aa342520a0ed8.svg
fmk
user
fmk99
68306ffdff038ca6400a153d
Shuai Zhang
68306ffdff038ca6400a153e
Zhengqi Wen
68306ffdff038ca6400a153f
Pengpeng Shao
68306ffdff038ca6400a1540
Huazhe Xu
68306ffdff038ca6400a1541
Jianhua Tao
Thought-Augmented Policy Optimization: Bridging External Guidance and
  Internal Capabilities
Reinforcement learning (RL) has emerged as an effective method for training
reasoning models. However, existing RL approaches typically bias the model's
output distribution toward reward-maximizing paths without introducing external
knowledge. This limits their exploration capacity and results in a narrower
reasoning capability boundary compared to base models. To address this
limitation, we propose TAPO (Thought-Augmented Policy Optimization), a novel
framework that augments RL by incorporating external high-level guidance
("thought patterns"). By adaptively integrating structured thoughts during
training, TAPO effectively balances model-internal exploration and external
guidance exploitation. Extensive experiments show that our approach
significantly outperforms GRPO by 99% on AIME, 41% on AMC, and 17% on Minerva
Math. Notably, these high-level thought patterns, abstracted from only 500
prior samples, generalize effectively across various tasks and models. This
highlights TAPO's potential for broader applications across multiple tasks and
domains. Our further analysis reveals that introducing external guidance
produces powerful reasoning models with superior explainability of inference
behavior and enhanced output readability.
A novel RL framework, TAPO, integrates external guidance to enhance model performance and exploration compared to existing methods.
reinforcement learning TAPO Thought-Augmented Policy Optimization high-level guidance thought patterns model exploration AIME AMC Minerva Math reasoning models explainability output readability