Scaling Autonomous Agents via Automatic Reward Modeling And Planning
Large language models (LLMs) have demonstrated remarkable capabilities across
a range of text-generation tasks. However, LLMs still struggle with problems
requiring multi-step decision-making and environmental feedback, such as online
shopping, scientific reasoning, and mathematical problem-solving. Unlike pure
text data, collecting large-scale decision-making data is challenging.
Moreover, many powerful LLMs are only accessible through APIs, which hinders
their fine-tuning for agent tasks due to cost and complexity. To address LLM
agents' limitations, we propose a framework that can automatically learn a
reward model from the environment without human annotations. This model can be
used to evaluate the action trajectories of LLM agents and provide heuristics
for task planning. Specifically, our approach involves employing one LLM-based
agent to navigate an environment randomly, generating diverse action
trajectories. Subsequently, a separate LLM is leveraged to assign a task intent
and synthesize a negative response alongside the correct response for each
trajectory. These triplets (task intent, positive response, and negative
response) are then utilized as training data to optimize a reward model capable
of scoring action trajectories. The effectiveness and generalizability of our
framework are demonstrated through evaluations conducted on different agent
benchmarks. In conclusion, our proposed framework represents a significant
advancement in enhancing LLM agents' decision-making capabilities. By
automating the learning of reward models, we overcome the challenges of data
scarcity and API limitations, potentially revolutionizing the application of
LLMs in complex and interactive environments. This research paves the way for
more sophisticated AI agents capable of tackling a wide range of real-world
problems requiring multi-step decision-making.
67b657d6a267b1a747a7fed6
Zhenfang Chen
67b657d6a267b1a747a7fed7
Delin Chen
claimed_verified
2025-04-13T19:27:46.426Z
62189e34523d13b0992f67ac
/avatars/b5153ff75b20185b195eb38c4786fd34.svg
Delin Chen
user
chendl
67b657d6a267b1a747a7fed8
Rui Sun
claimed_verified
2025-02-20T09:36:34.773Z
654e024de113b04ba5c71e2f
https://cdn-avatars.huggingface.co/v1/production/uploads/654e024de113b04ba5c71e2f/WH6S_gpQU6OXqDaiPpheK.jpeg
Rui Sun
user
ThreeSR
67b657d6a267b1a747a7fed9
Wenjun Liu
67b657d6a267b1a747a7feda
Chuang Gan
Scaling Autonomous Agents via Automatic Reward Modeling And Planning
Large language models (LLMs) have demonstrated remarkable capabilities across
a range of text-generation tasks. However, LLMs still struggle with problems
requiring multi-step decision-making and environmental feedback, such as online
shopping, scientific reasoning, and mathematical problem-solving. Unlike pure
text data, collecting large-scale decision-making data is challenging.
Moreover, many powerful LLMs are only accessible through APIs, which hinders
their fine-tuning for agent tasks due to cost and complexity. To address LLM
agents' limitations, we propose a framework that can automatically learn a
reward model from the environment without human annotations. This model can be
used to evaluate the action trajectories of LLM agents and provide heuristics
for task planning. Specifically, our approach involves employing one LLM-based
agent to navigate an environment randomly, generating diverse action
trajectories. Subsequently, a separate LLM is leveraged to assign a task intent
and synthesize a negative response alongside the correct response for each
trajectory. These triplets (task intent, positive response, and negative
response) are then utilized as training data to optimize a reward model capable
of scoring action trajectories. The effectiveness and generalizability of our
framework are demonstrated through evaluations conducted on different agent
benchmarks. In conclusion, our proposed framework represents a significant
advancement in enhancing LLM agents' decision-making capabilities. By
automating the learning of reward models, we overcome the challenges of data
scarcity and API limitations, potentially revolutionizing the application of
LLMs in complex and interactive environments. This research paves the way for
more sophisticated AI agents capable of tackling a wide range of real-world
problems requiring multi-step decision-making.
A framework is proposed to automate the learning of reward models for LLM agents, enhancing their decision-making abilities in complex environments through synthetic data generation.
LLMs large language models multi-step decision-making environmental feedback task intent action trajectories reward model synthetic data agent benchmarks