title: GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based
    VLM Agent Training
summary: Reinforcement learning with verifiable outcome rewards (RLVR) has effectively
  scaled up chain-of-thought (CoT) reasoning in large language models (LLMs).
  Yet, its efficacy in training vision-language model (VLM) agents for
  goal-directed action reasoning in visual environments is less established. This
  work investigates this problem through extensive experiments on complex card
  games, such as 24 points, and embodied tasks from ALFWorld. We find that when
  rewards are based solely on action outcomes, RL fails to incentivize CoT
  reasoning in VLMs, instead leading to a phenomenon we termed thought collapse,
  characterized by a rapid loss of diversity in the agent's thoughts,
  state-irrelevant and incomplete reasoning, and subsequent invalid actions,
  resulting in negative rewards. To counteract thought collapse, we highlight the
  necessity of process guidance and propose an automated corrector that evaluates
  and refines the agent's reasoning at each RL step. This simple and scalable GTR
  (Guided Thought Reinforcement) framework trains reasoning and action
  simultaneously without the need for dense, per-step human labeling. Our
  experiments demonstrate that GTR significantly enhances the performance and
  generalization of the LLaVA-7b model across various visual environments,
  achieving 3-5 times higher task success rates compared to SoTA models with
  notably smaller model sizes.
paper_authors._id: 67d280f20a6a6dd4a0ffe9e8
paper_authors.name: Tong Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-14T08:58:47.531Z
paper_authors._id: 66c2e8b4a03b764ca9057e65
paper_authors.avatarUrl: /avatars/ce17d68f02a08c148ebb9df3170812c8.svg
paper_authors.fullname: Tong Wei
paper_authors.type: user
paper_authors.user: weit123
paper_authors._id: 67d280f20a6a6dd4a0ffe9e9
paper_authors.name: Yijun Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:23:38.382Z
paper_authors._id: 645e553c3b6d85c65e8b0e54
paper_authors.avatarUrl: /avatars/1fffc6499b9d65b21a895ca96f03b781.svg
paper_authors.fullname: Steven
paper_authors.type: user
paper_authors.user: yijunyang
paper_authors._id: 67d280f20a6a6dd4a0ffe9ea
paper_authors.name: Junliang Xing
paper_authors._id: 67d280f20a6a6dd4a0ffe9eb
paper_authors.name: Yuanchun Shi
paper_authors._id: 67d280f20a6a6dd4a0ffe9ec
paper_authors.name: Zongqing Lu
paper_authors._id: 67d280f20a6a6dd4a0ffe9ed
paper_authors.name: Deheng Ye
paper_title: GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based
    VLM Agent Training
paper_summary: Reinforcement learning with verifiable outcome rewards (RLVR) has effectively
  scaled up chain-of-thought (CoT) reasoning in large language models (LLMs).
  Yet, its efficacy in training vision-language model (VLM) agents for
  goal-directed action reasoning in visual environments is less established. This
  work investigates this problem through extensive experiments on complex card
  games, such as 24 points, and embodied tasks from ALFWorld. We find that when
  rewards are based solely on action outcomes, RL fails to incentivize CoT
  reasoning in VLMs, instead leading to a phenomenon we termed thought collapse,
  characterized by a rapid loss of diversity in the agent's thoughts,
  state-irrelevant and incomplete reasoning, and subsequent invalid actions,
  resulting in negative rewards. To counteract thought collapse, we highlight the
  necessity of process guidance and propose an automated corrector that evaluates
  and refines the agent's reasoning at each RL step. This simple and scalable GTR
  (Guided Thought Reinforcement) framework trains reasoning and action
  simultaneously without the need for dense, per-step human labeling. Our
  experiments demonstrate that GTR significantly enhances the performance and
  generalization of the LLaVA-7b model across various visual environments,
  achieving 3-5 times higher task success rates compared to SoTA models with
  notably smaller model sizes.
paper_ai_summary: GTR framework enhances VLM agent performance in visual environments by addressing thought collapse through process guidance and automated reasoning refinement.
paper_ai_keywords: Reinforcement learning verifiable outcome rewards RLVR chain-of-thought reasoning large language models vision-language models goal-directed action reasoning thought collapse state-irrelevant reasoning invalid actions negative rewards process guidance Guided Thought Reinforcement GTR LLaVA-7b task success rates