title: Thought-Augmented Policy Optimization: Bridging External Guidance and
    Internal Capabilities
summary: Reinforcement learning (RL) has emerged as an effective method for training
  reasoning models. However, existing RL approaches typically bias the model's
  output distribution toward reward-maximizing paths without introducing external
  knowledge. This limits their exploration capacity and results in a narrower
  reasoning capability boundary compared to base models. To address this
  limitation, we propose TAPO (Thought-Augmented Policy Optimization), a novel
  framework that augments RL by incorporating external high-level guidance
  ("thought patterns"). By adaptively integrating structured thoughts during
  training, TAPO effectively balances model-internal exploration and external
  guidance exploitation. Extensive experiments show that our approach
  significantly outperforms GRPO by 99% on AIME, 41% on AMC, and 17% on Minerva
  Math. Notably, these high-level thought patterns, abstracted from only 500
  prior samples, generalize effectively across various tasks and models. This
  highlights TAPO's potential for broader applications across multiple tasks and
  domains. Our further analysis reveals that introducing external guidance
  produces powerful reasoning models with superior explainability of inference
  behavior and enhanced output readability.
paper_authors._id: 68306ffdff038ca6400a153a
paper_authors.name: Jinyang Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:13:57.397Z
paper_authors._id: 6747de57f8cab58c22ec94a2
paper_authors.avatarUrl: /avatars/5bae0341862fac24564781c0fa32aac5.svg
paper_authors.fullname: Jinyang Wu
paper_authors.type: user
paper_authors.user: Jinyang23
paper_authors._id: 68306ffdff038ca6400a153b
paper_authors.name: Chonghua Liao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:16:00.056Z
paper_authors._id: 667fdaee20ee9ac417c7708c
paper_authors.avatarUrl: /avatars/69dfba6ff392643af1dcfe8af0a42ae9.svg
paper_authors.fullname: Chonghua Liao
paper_authors.type: user
paper_authors.user: ChonghuaLiao
paper_authors._id: 68306ffdff038ca6400a153c
paper_authors.name: Mingkuan Feng
paper_authors._id: 68306ffdff038ca6400a153d
paper_authors.name: Shuai Zhang
paper_authors._id: 68306ffdff038ca6400a153e
paper_authors.name: Zhengqi Wen
paper_authors._id: 68306ffdff038ca6400a153f
paper_authors.name: Pengpeng Shao
paper_authors._id: 68306ffdff038ca6400a1540
paper_authors.name: Huazhe Xu
paper_authors._id: 68306ffdff038ca6400a1541
paper_authors.name: Jianhua Tao
paper_title: Thought-Augmented Policy Optimization: Bridging External Guidance and
    Internal Capabilities
paper_summary: Reinforcement learning (RL) has emerged as an effective method for training
  reasoning models. However, existing RL approaches typically bias the model's
  output distribution toward reward-maximizing paths without introducing external
  knowledge. This limits their exploration capacity and results in a narrower
  reasoning capability boundary compared to base models. To address this
  limitation, we propose TAPO (Thought-Augmented Policy Optimization), a novel
  framework that augments RL by incorporating external high-level guidance
  ("thought patterns"). By adaptively integrating structured thoughts during
  training, TAPO effectively balances model-internal exploration and external
  guidance exploitation. Extensive experiments show that our approach
  significantly outperforms GRPO by 99% on AIME, 41% on AMC, and 17% on Minerva
  Math. Notably, these high-level thought patterns, abstracted from only 500
  prior samples, generalize effectively across various tasks and models. This
  highlights TAPO's potential for broader applications across multiple tasks and
  domains. Our further analysis reveals that introducing external guidance
  produces powerful reasoning models with superior explainability of inference
  behavior and enhanced output readability.
paper_ai_summary: A novel RL framework, TAPO, integrates external guidance to enhance model performance and exploration compared to existing methods.
paper_ai_keywords: reinforcement learning TAPO Thought-Augmented Policy Optimization high-level guidance thought patterns model exploration AIME AMC Minerva Math reasoning models explainability output readability