title: Scaling Autonomous Agents via Automatic Reward Modeling And Planning
summary: Large language models (LLMs) have demonstrated remarkable capabilities across
  a range of text-generation tasks. However, LLMs still struggle with problems
  requiring multi-step decision-making and environmental feedback, such as online
  shopping, scientific reasoning, and mathematical problem-solving. Unlike pure
  text data, collecting large-scale decision-making data is challenging.
  Moreover, many powerful LLMs are only accessible through APIs, which hinders
  their fine-tuning for agent tasks due to cost and complexity. To address LLM
  agents' limitations, we propose a framework that can automatically learn a
  reward model from the environment without human annotations. This model can be
  used to evaluate the action trajectories of LLM agents and provide heuristics
  for task planning. Specifically, our approach involves employing one LLM-based
  agent to navigate an environment randomly, generating diverse action
  trajectories. Subsequently, a separate LLM is leveraged to assign a task intent
  and synthesize a negative response alongside the correct response for each
  trajectory. These triplets (task intent, positive response, and negative
  response) are then utilized as training data to optimize a reward model capable
  of scoring action trajectories. The effectiveness and generalizability of our
  framework are demonstrated through evaluations conducted on different agent
  benchmarks. In conclusion, our proposed framework represents a significant
  advancement in enhancing LLM agents' decision-making capabilities. By
  automating the learning of reward models, we overcome the challenges of data
  scarcity and API limitations, potentially revolutionizing the application of
  LLMs in complex and interactive environments. This research paves the way for
  more sophisticated AI agents capable of tackling a wide range of real-world
  problems requiring multi-step decision-making.
paper_authors._id: 67b657d6a267b1a747a7fed6
paper_authors.name: Zhenfang Chen
paper_authors._id: 67b657d6a267b1a747a7fed7
paper_authors.name: Delin Chen
paper_authors._id: 67b657d6a267b1a747a7fed8
paper_authors.name: Rui Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T09:36:34.773Z
paper_authors._id: 654e024de113b04ba5c71e2f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654e024de113b04ba5c71e2f/WH6S_gpQU6OXqDaiPpheK.jpeg
paper_authors.fullname: Rui Sun
paper_authors.type: user
paper_authors.user: ThreeSR
paper_authors._id: 67b657d6a267b1a747a7fed9
paper_authors.name: Wenjun Liu
paper_authors._id: 67b657d6a267b1a747a7feda
paper_authors.name: Chuang Gan
paper_title: Scaling Autonomous Agents via Automatic Reward Modeling And Planning
paper_summary: Large language models (LLMs) have demonstrated remarkable capabilities across
  a range of text-generation tasks. However, LLMs still struggle with problems
  requiring multi-step decision-making and environmental feedback, such as online
  shopping, scientific reasoning, and mathematical problem-solving. Unlike pure
  text data, collecting large-scale decision-making data is challenging.
  Moreover, many powerful LLMs are only accessible through APIs, which hinders
  their fine-tuning for agent tasks due to cost and complexity. To address LLM
  agents' limitations, we propose a framework that can automatically learn a
  reward model from the environment without human annotations. This model can be
  used to evaluate the action trajectories of LLM agents and provide heuristics
  for task planning. Specifically, our approach involves employing one LLM-based
  agent to navigate an environment randomly, generating diverse action
  trajectories. Subsequently, a separate LLM is leveraged to assign a task intent
  and synthesize a negative response alongside the correct response for each
  trajectory. These triplets (task intent, positive response, and negative
  response) are then utilized as training data to optimize a reward model capable
  of scoring action trajectories. The effectiveness and generalizability of our
  framework are demonstrated through evaluations conducted on different agent
  benchmarks. In conclusion, our proposed framework represents a significant
  advancement in enhancing LLM agents' decision-making capabilities. By
  automating the learning of reward models, we overcome the challenges of data
  scarcity and API limitations, potentially revolutionizing the application of
  LLMs in complex and interactive environments. This research paves the way for
  more sophisticated AI agents capable of tackling a wide range of real-world
  problems requiring multi-step decision-making.
paper_ai_keywords: reward model action trajectories task intent positive response negative response LLM agents API limitations data scarcity