AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
We introduce AgentSynth, a scalable and cost-efficient pipeline for
automatically synthesizing high-quality tasks and trajectory datasets for
generalist computer-use agents. Leveraging information asymmetry, AgentSynth
constructs subtasks that are simple during generation but significantly more
challenging when composed into long-horizon tasks, enabling the creation of
over 6,000 diverse and realistic tasks. Our pipeline begins with an LLM-based
task proposer guided by a persona, followed by an execution agent that
completes the task and logs the trajectory. This process is repeated
iteratively to form a sequence of subtasks, which are then summarized by a
separate agent into a composite task of controllable difficulty. A key strength
of AgentSynth is its ability to precisely modulate task complexity by varying
the number of subtasks. Empirical evaluations show that state-of-the-art LLM
agents suffer a steep performance drop, from 18% success at difficulty level 1
to just 4% at level 6, highlighting the benchmark's difficulty and
discriminative power. Moreover, our pipeline achieves a low average cost of
\$0.60 per trajectory, orders of magnitude cheaper than human annotations. Our
code and data are publicly available at
https://github.com/sunblaze-ucb/AgentSynth
6853647b99bf39f9665c7941
Jingxu Xie
6853647b99bf39f9665c7942
Dylan Xu
6853647b99bf39f9665c7943
Xuandong Zhao
6853647b99bf39f9665c7944
Dawn Song
AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
We introduce AgentSynth, a scalable and cost-efficient pipeline for
automatically synthesizing high-quality tasks and trajectory datasets for
generalist computer-use agents. Leveraging information asymmetry, AgentSynth
constructs subtasks that are simple during generation but significantly more
challenging when composed into long-horizon tasks, enabling the creation of
over 6,000 diverse and realistic tasks. Our pipeline begins with an LLM-based
task proposer guided by a persona, followed by an execution agent that
completes the task and logs the trajectory. This process is repeated
iteratively to form a sequence of subtasks, which are then summarized by a
separate agent into a composite task of controllable difficulty. A key strength
of AgentSynth is its ability to precisely modulate task complexity by varying
the number of subtasks. Empirical evaluations show that state-of-the-art LLM
agents suffer a steep performance drop, from 18% success at difficulty level 1
to just 4% at level 6, highlighting the benchmark's difficulty and
discriminative power. Moreover, our pipeline achieves a low average cost of
\$0.60 per trajectory, orders of magnitude cheaper than human annotations. Our
code and data are publicly available at
https://github.com/sunblaze-ucb/AgentSynth
AgentSynth synthesizes high-quality, diverse tasks and trajectory datasets for generalist computer-use agents using LLMs and an iterative subtask construction approach, enabling precise control over task complexity and offering significant cost savings compared to human annotations.
LLM-based task proposer execution agent composite task subtasks state-of-the-art LLM agents performance drop human annotations