title: Agent-R: Training Language Model Agents to Reflect via Iterative
    Self-Training
summary: Large Language Models (LLMs) agents are increasingly pivotal for addressing
  complex tasks in interactive environments. Existing work mainly focuses on
  enhancing performance through behavior cloning from stronger experts, yet such
  approaches often falter in real-world applications, mainly due to the inability
  to recover from errors. However, step-level critique data is difficult and
  expensive to collect. Automating and dynamically constructing self-critique
  datasets is thus crucial to empowering models with intelligent agent
  capabilities. In this work, we propose an iterative self-training framework,
  Agent-R, that enables language Agent to Reflect on the fly. Unlike traditional
  methods that reward or penalize actions based on correctness, Agent-R leverages
  MCTS to construct training data that recover correct trajectories from
  erroneous ones. A key challenge of agent reflection lies in the necessity for
  timely revision rather than waiting until the end of a rollout. To address
  this, we introduce a model-guided critique construction mechanism: the actor
  model identifies the first error step (within its current capability) in a
  failed trajectory. Starting from it, we splice it with the adjacent correct
  path, which shares the same parent node in the tree. This strategy enables the
  model to learn reflection based on its current policy, therefore yielding
  better learning efficiency. To further explore the scalability of this
  self-improvement paradigm, we investigate iterative refinement of both error
  correction capabilities and dataset construction. Our findings demonstrate that
  Agent-R continuously improves the model's ability to recover from errors and
  enables timely error correction. Experiments on three interactive environments
  show that Agent-R effectively equips agents to correct erroneous actions while
  avoiding loops, achieving superior performance compared to baseline methods
  (+5.59%).
paper_authors._id: 679080298ad1d8203a994f7f
paper_authors.name: Siyu Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T11:00:12.151Z
paper_authors._id: 62d62b333bf5e059f7d2b286
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668513815771-62d62b333bf5e059f7d2b286.jpeg
paper_authors.fullname: Siyu Yuan
paper_authors.type: user
paper_authors.user: siyuyuan
paper_authors._id: 679080298ad1d8203a994f80
paper_authors.name: Zehui Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T11:00:20.946Z
paper_authors._id: 64892d31cbda0d1cdb956897
paper_authors.avatarUrl: /avatars/3cdafe03a8295124636347d15a099aaf.svg
paper_authors.fullname: Zehui Chen
paper_authors.type: user
paper_authors.user: lovesnowbest
paper_authors._id: 679080298ad1d8203a994f81
paper_authors.name: Zhiheng Xi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T14:04:23.785Z
paper_authors._id: 653a6e5cae155b92bae77b74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
paper_authors.fullname: Zhiheng Xi
paper_authors.type: user
paper_authors.user: WooooDyy
paper_authors._id: 679080298ad1d8203a994f82
paper_authors.name: Junjie Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-22T10:08:02.519Z
paper_authors._id: 66384be673c2c55f2ded89fa
paper_authors.avatarUrl: /avatars/1d8721074f0f51fab405f81474f2035f.svg
paper_authors.fullname: Junjie Ye
paper_authors.type: user
paper_authors.user: Junjie-Ye
paper_authors._id: 679080298ad1d8203a994f83
paper_authors.name: Zhengyin Du
paper_authors._id: 679080298ad1d8203a994f84
paper_authors.name: Jiecao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T14:04:37.918Z
paper_authors._id: 66df70fe5a0c5910d663160d
paper_authors.avatarUrl: /avatars/980ca32bd0049ef5bbf002e7dc9f911c.svg
paper_authors.fullname: jiecao.chen
paper_authors.type: user
paper_authors.user: xmerge123
paper_title: Agent-R: Training Language Model Agents to Reflect via Iterative
    Self-Training
paper_summary: Large Language Models (LLMs) agents are increasingly pivotal for addressing
  complex tasks in interactive environments. Existing work mainly focuses on
  enhancing performance through behavior cloning from stronger experts, yet such
  approaches often falter in real-world applications, mainly due to the inability
  to recover from errors. However, step-level critique data is difficult and
  expensive to collect. Automating and dynamically constructing self-critique
  datasets is thus crucial to empowering models with intelligent agent
  capabilities. In this work, we propose an iterative self-training framework,
  Agent-R, that enables language Agent to Reflect on the fly. Unlike traditional
  methods that reward or penalize actions based on correctness, Agent-R leverages
  MCTS to construct training data that recover correct trajectories from
  erroneous ones. A key challenge of agent reflection lies in the necessity for
  timely revision rather than waiting until the end of a rollout. To address
  this, we introduce a model-guided critique construction mechanism: the actor
  model identifies the first error step (within its current capability) in a
  failed trajectory. Starting from it, we splice it with the adjacent correct
  path, which shares the same parent node in the tree. This strategy enables the
  model to learn reflection based on its current policy, therefore yielding
  better learning efficiency. To further explore the scalability of this
  self-improvement paradigm, we investigate iterative refinement of both error
  correction capabilities and dataset construction. Our findings demonstrate that
  Agent-R continuously improves the model's ability to recover from errors and
  enables timely error correction. Experiments on three interactive environments
  show that Agent-R effectively equips agents to correct erroneous actions while
  avoiding loops, achieving superior performance compared to baseline methods
  (+5.59%).
paper_ai_keywords: Large Language Models (LLMs) dense language models behavior cloning step-level critique data self-critique datasets iterative self-training framework Agent-R MCTS (Monte Carlo Tree Search) feature agent reflection reward or penalize actions timely revision actor model correct trajectories error correction capabilities iterative refinement learning efficiency error recovery timely error correction interactive environments baseline methods