title: AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web
    Tutorials
summary: Graphical User Interface (GUI) agents hold great potential for automating
  complex tasks across diverse digital environments, from web applications to
  desktop software. However, the development of such agents is hindered by the
  lack of high-quality, multi-step trajectory data required for effective
  training. Existing approaches rely on expensive and labor-intensive human
  annotation, making them unsustainable at scale. To address this challenge, we
  propose AgentTrek, a scalable data synthesis pipeline that generates
  high-quality GUI agent trajectories by leveraging web tutorials. Our method
  automatically gathers tutorial-like texts from the internet, transforms them
  into task goals with step-by-step instructions, and employs a visual-language
  model agent to simulate their execution in a real digital environment. A
  VLM-based evaluator ensures the correctness of the generated trajectories. We
  demonstrate that training GUI agents with these synthesized trajectories
  significantly improves their grounding and planning performance over the
  current models. Moreover, our approach is more cost-efficient compared to
  traditional human annotation methods. This work underscores the potential of
  guided replay with web tutorials as a viable strategy for large-scale GUI agent
  training, paving the way for more capable and autonomous digital agents.
paper_authors._id: 675bcc75b497da95e5a4ffd1
paper_authors.name: Yiheng Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:15:41.823Z
paper_authors._id: 601d29ab913ad3afd7b7ddb8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620447944896-601d29ab913ad3afd7b7ddb8.jpeg
paper_authors.fullname: Yiheng Xu
paper_authors.type: user
paper_authors.user: ranpox
paper_authors._id: 675bcc75b497da95e5a4ffd2
paper_authors.name: Dunjie Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T09:53:35.449Z
paper_authors._id: 669ca7e678115e16bdfc9bfc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/669ca7e678115e16bdfc9bfc/pku8NvQKqfNQACRqm1YrW.jpeg
paper_authors.fullname: Lu Dunjie
paper_authors.type: user
paper_authors.user: ludunjie
paper_authors._id: 675bcc75b497da95e5a4ffd3
paper_authors.name: Zhennan Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:11:47.637Z
paper_authors._id: 64bfa52094c0e3be4a33863e
paper_authors.avatarUrl: /avatars/bc0737bf4481433b669433facf6ce6f4.svg
paper_authors.fullname: Shenzhennan
paper_authors.type: user
paper_authors.user: 5456es
paper_authors._id: 675bcc75b497da95e5a4ffd4
paper_authors.name: Junli Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T08:56:03.169Z
paper_authors._id: 65f944d5056d465a38f49361
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/PWp43VsXeltiPifGxGwrn.jpeg
paper_authors.fullname: Junli Wang
paper_authors.type: user
paper_authors.user: ZeonLap
paper_authors._id: 675bcc75b497da95e5a4ffd5
paper_authors.name: Zekun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:15:40.064Z
paper_authors._id: 656832dfbd65fd41ee7aa8cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
paper_authors.fullname: Zekun Wang
paper_authors.type: user
paper_authors.user: kugwzk
paper_authors._id: 675bcc75b497da95e5a4ffd6
paper_authors.name: Yuchen Mao
paper_authors._id: 675bcc75b497da95e5a4ffd7
paper_authors.name: Caiming Xiong
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-13T05:56:07.585Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_authors._id: 675bcc75b497da95e5a4ffd8
paper_authors.name: Tao Yu
paper_title: AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web
    Tutorials
paper_summary: Graphical User Interface (GUI) agents hold great potential for automating
  complex tasks across diverse digital environments, from web applications to
  desktop software. However, the development of such agents is hindered by the
  lack of high-quality, multi-step trajectory data required for effective
  training. Existing approaches rely on expensive and labor-intensive human
  annotation, making them unsustainable at scale. To address this challenge, we
  propose AgentTrek, a scalable data synthesis pipeline that generates
  high-quality GUI agent trajectories by leveraging web tutorials. Our method
  automatically gathers tutorial-like texts from the internet, transforms them
  into task goals with step-by-step instructions, and employs a visual-language
  model agent to simulate their execution in a real digital environment. A
  VLM-based evaluator ensures the correctness of the generated trajectories. We
  demonstrate that training GUI agents with these synthesized trajectories
  significantly improves their grounding and planning performance over the
  current models. Moreover, our approach is more cost-efficient compared to
  traditional human annotation methods. This work underscores the potential of
  guided replay with web tutorials as a viable strategy for large-scale GUI agent
  training, paving the way for more capable and autonomous digital agents.
paper_ai_summary: AgentTrek synthesizes high-quality GUI agent trajectories using web tutorials, improving their performance cost-effectively compared to human annotation.
paper_ai_keywords: gui agents data synthesis pipeline agenttrek web tutorials task goals step-by-step instructions visual-language model visual-language model agent evaluator cost-efficient grounded planning performance