title: Exploring Expert Failures Improves LLM Agent Tuning
summary: Large Language Models (LLMs) have shown tremendous potential as agents,
  excelling at tasks that require multiple rounds of reasoning and interactions.
  Rejection Sampling Fine-Tuning (RFT) has emerged as an effective method for
  finetuning LLMs as agents: it first imitates expert-generated successful
  trajectories and further improves agentic skills through iterative fine-tuning
  on successful, self-generated trajectories. However, since the expert (e.g.,
  GPT-4) succeeds primarily on simpler subtasks and RFT inherently favors simpler
  scenarios, many complex subtasks remain unsolved and persistently
  out-of-distribution (OOD). Upon investigating these challenging subtasks, we
  discovered that previously failed expert trajectories can often provide
  valuable guidance, e.g., plans and key actions, that can significantly improve
  agent exploration efficiency and acquisition of critical skills. Motivated by
  these observations, we propose Exploring Expert Failures (EEF), which
  identifies beneficial actions from failed expert trajectories and integrates
  them into the training dataset. Potentially harmful actions are meticulously
  excluded to prevent contamination of the model learning process. By leveraging
  the beneficial actions in expert failures, EEF successfully solves some
  previously unsolvable subtasks and improves agent tuning performance.
  Remarkably, our approach achieved a 62\% win rate in WebShop, outperforming RFT
  (53. 6\%) and GPT-4 (35. 6\%), and to the best of our knowledge, setting a new
  state-of-the-art as the first method to surpass a score of 0.81 in WebShop and
  exceed 81 in SciWorld.
paper_authors._id: 6801cbcc382250483109ddd4
paper_authors.name: Li-Cheng Lan
paper_authors._id: 6801cbcc382250483109ddd5
paper_authors.name: Andrew Bai
paper_authors._id: 6801cbcc382250483109ddd6
paper_authors.name: Minhao Cheng
paper_authors._id: 6801cbcc382250483109ddd8
paper_authors.name: Cho-Jui Hsieh
paper_authors._id: 6801cbcc382250483109ddd9
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-18T09:38:55.704Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_title: Exploring Expert Failures Improves LLM Agent Tuning
paper_summary: Large Language Models (LLMs) have shown tremendous potential as agents,
  excelling at tasks that require multiple rounds of reasoning and interactions.
  Rejection Sampling Fine-Tuning (RFT) has emerged as an effective method for
  finetuning LLMs as agents: it first imitates expert-generated successful
  trajectories and further improves agentic skills through iterative fine-tuning
  on successful, self-generated trajectories. However, since the expert (e.g.,
  GPT-4) succeeds primarily on simpler subtasks and RFT inherently favors simpler
  scenarios, many complex subtasks remain unsolved and persistently
  out-of-distribution (OOD). Upon investigating these challenging subtasks, we
  discovered that previously failed expert trajectories can often provide
  valuable guidance, e.g., plans and key actions, that can significantly improve
  agent exploration efficiency and acquisition of critical skills. Motivated by
  these observations, we propose Exploring Expert Failures (EEF), which
  identifies beneficial actions from failed expert trajectories and integrates
  them into the training dataset. Potentially harmful actions are meticulously
  excluded to prevent contamination of the model learning process. By leveraging
  the beneficial actions in expert failures, EEF successfully solves some
  previously unsolvable subtasks and improves agent tuning performance.
  Remarkably, our approach achieved a 62\% win rate in WebShop, outperforming RFT
  (53. 6\%) and GPT-4 (35. 6\%), and to the best of our knowledge, setting a new
  state-of-the-art as the first method to surpass a score of 0.81 in WebShop and
  exceed 81 in SciWorld.
paper_ai_summary: A new method, Exploring Expert Failures, enhances fine-tuning of Large Language Models by incorporating beneficial actions from failed expert trajectories, significantly improving performance in complex tasks like WebShop and SciWorld.
paper_ai_keywords: Large Language Models RFT fine-tuning expert trajectories rejection sampling exploratory expert failures subtasks OOD agentic skills WebShop SciWorld