title: Towards Internet-Scale Training For Agents
summary: The predominant approach for training web navigation agents gathers human
  demonstrations for a set of popular websites and hand-written tasks, but it is
  becoming clear that human data are an inefficient resource. We develop a
  pipeline to facilitate Internet-scale training for agents without laborious
  human annotations. In the first stage, an LLM generates tasks for 150k diverse
  websites. In the next stage, LLM agents complete tasks and produce
  trajectories. In the final stage, an LLM reviews the trajectories and judges
  their success. Language models are competitive with human annotators, detecting
  and filtering out harmful content with an accuracy of 97%, generating feasible
  tasks with an 89% rate, and judging successful trajectories with an 82.6%
  accuracy. Scaling the pipeline, agents based on Llama 3.1 70B solve 16.7% of
  tasks for 150k sites. Training on the data generated by our pipeline is
  competitive with training on human demonstrations. In data-limited settings
  derived from Mind2Web and WebLINX, we improve Step Accuracy by up to +89.5% and
  +122.1% respectively for agents trained on mixtures of data from our pipeline,
  and human data. When training agents with all available human data from these
  benchmarks, agents fail to generalize to diverse real sites, and adding our
  data improves their generalization by +149.0% for WebLINX and +156.3% for
  Mind2Web. Code will be available at: data-for-agents.github.io.
paper_authors._id: 67aae05bb893603a0b4b241d
paper_authors.name: Brandon Trabucco
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T17:37:50.173Z
paper_authors._id: 632d8b2e1d8a018adf4f98f1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632d8b2e1d8a018adf4f98f1/vGvpkxyGLNQSJmEONR2uX.jpeg
paper_authors.fullname: Brandon Trabucco
paper_authors.type: user
paper_authors.user: btrabucco
paper_authors._id: 67aae05bb893603a0b4b241e
paper_authors.name: Gunnar Sigurdsson
paper_authors._id: 67aae05bb893603a0b4b241f
paper_authors.name: Robinson Piramuthu
paper_authors._id: 67aae05bb893603a0b4b2420
paper_authors.name: Ruslan Salakhutdinov
paper_title: Towards Internet-Scale Training For Agents
paper_summary: The predominant approach for training web navigation agents gathers human
  demonstrations for a set of popular websites and hand-written tasks, but it is
  becoming clear that human data are an inefficient resource. We develop a
  pipeline to facilitate Internet-scale training for agents without laborious
  human annotations. In the first stage, an LLM generates tasks for 150k diverse
  websites. In the next stage, LLM agents complete tasks and produce
  trajectories. In the final stage, an LLM reviews the trajectories and judges
  their success. Language models are competitive with human annotators, detecting
  and filtering out harmful content with an accuracy of 97%, generating feasible
  tasks with an 89% rate, and judging successful trajectories with an 82.6%
  accuracy. Scaling the pipeline, agents based on Llama 3.1 70B solve 16.7% of
  tasks for 150k sites. Training on the data generated by our pipeline is
  competitive with training on human demonstrations. In data-limited settings
  derived from Mind2Web and WebLINX, we improve Step Accuracy by up to +89.5% and
  +122.1% respectively for agents trained on mixtures of data from our pipeline,
  and human data. When training agents with all available human data from these
  benchmarks, agents fail to generalize to diverse real sites, and adding our
  data improves their generalization by +149.0% for WebLINX and +156.3% for
  Mind2Web. Code will be available at: data-for-agents.github.io.
paper_ai_keywords: LLM LLM agents trajectories LLM Llama 3.1 70B Step Accuracy Mind2Web WebLINX Real sites