AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web
  Tutorials
Graphical User Interface (GUI) agents hold great potential for automating
complex tasks across diverse digital environments, from web applications to
desktop software. However, the development of such agents is hindered by the
lack of high-quality, multi-step trajectory data required for effective
training. Existing approaches rely on expensive and labor-intensive human
annotation, making them unsustainable at scale. To address this challenge, we
propose AgentTrek, a scalable data synthesis pipeline that generates
high-quality GUI agent trajectories by leveraging web tutorials. Our method
automatically gathers tutorial-like texts from the internet, transforms them
into task goals with step-by-step instructions, and employs a visual-language
model agent to simulate their execution in a real digital environment. A
VLM-based evaluator ensures the correctness of the generated trajectories. We
demonstrate that training GUI agents with these synthesized trajectories
significantly improves their grounding and planning performance over the
current models. Moreover, our approach is more cost-efficient compared to
traditional human annotation methods. This work underscores the potential of
guided replay with web tutorials as a viable strategy for large-scale GUI agent
training, paving the way for more capable and autonomous digital agents.
675bcc75b497da95e5a4ffd1
Yiheng Xu
claimed_verified
2024-12-13T15:15:41.823Z
601d29ab913ad3afd7b7ddb8
https://cdn-avatars.huggingface.co/v1/production/uploads/1620447944896-601d29ab913ad3afd7b7ddb8.jpeg
Yiheng Xu
user
ranpox
675bcc75b497da95e5a4ffd2
Dunjie Lu
claimed_verified
2024-12-18T09:53:35.449Z
669ca7e678115e16bdfc9bfc
https://cdn-avatars.huggingface.co/v1/production/uploads/669ca7e678115e16bdfc9bfc/pku8NvQKqfNQACRqm1YrW.jpeg
Lu Dunjie
user
ludunjie
675bcc75b497da95e5a4ffd3
Zhennan Shen
claimed_verified
2024-12-23T11:11:47.637Z
64bfa52094c0e3be4a33863e
/avatars/bc0737bf4481433b669433facf6ce6f4.svg
Shenzhennan
user
5456es
675bcc75b497da95e5a4ffd4
Junli Wang
claimed_verified
2024-12-13T08:56:03.169Z
65f944d5056d465a38f49361
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/PWp43VsXeltiPifGxGwrn.jpeg
Junli Wang
user
ZeonLap
675bcc75b497da95e5a4ffd5
Zekun Wang
claimed_verified
2024-12-13T15:15:40.064Z
656832dfbd65fd41ee7aa8cd
https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
Zekun Wang
user
kugwzk
675bcc75b497da95e5a4ffd6
Yuchen Mao
675bcc75b497da95e5a4ffd7
Caiming Xiong
extracted_pending
2024-12-13T05:56:07.585Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
675bcc75b497da95e5a4ffd8
Tao Yu
AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web
  Tutorials
Graphical User Interface (GUI) agents hold great potential for automating
complex tasks across diverse digital environments, from web applications to
desktop software. However, the development of such agents is hindered by the
lack of high-quality, multi-step trajectory data required for effective
training. Existing approaches rely on expensive and labor-intensive human
annotation, making them unsustainable at scale. To address this challenge, we
propose AgentTrek, a scalable data synthesis pipeline that generates
high-quality GUI agent trajectories by leveraging web tutorials. Our method
automatically gathers tutorial-like texts from the internet, transforms them
into task goals with step-by-step instructions, and employs a visual-language
model agent to simulate their execution in a real digital environment. A
VLM-based evaluator ensures the correctness of the generated trajectories. We
demonstrate that training GUI agents with these synthesized trajectories
significantly improves their grounding and planning performance over the
current models. Moreover, our approach is more cost-efficient compared to
traditional human annotation methods. This work underscores the potential of
guided replay with web tutorials as a viable strategy for large-scale GUI agent
training, paving the way for more capable and autonomous digital agents.
AgentTrek synthesizes high-quality GUI agent trajectories using web tutorials, improving their performance cost-effectively compared to human annotation.
gui agents data synthesis pipeline agenttrek web tutorials task goals step-by-step instructions visual-language model visual-language model agent evaluator cost-efficient grounded planning performance