title: WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents
summary: In the realm of web agent research, achieving both generalization and
  accuracy remains a challenging problem. Due to high variance in website
  structure, existing approaches often fail. Moreover, existing fine-tuning and
  in-context learning techniques fail to generalize across multiple websites. We
  introduce Wilbur, an approach that uses a differentiable ranking model and a
  novel instruction synthesis technique to optimally populate a black-box large
  language model's prompt with task demonstrations from previous runs. To
  maximize end-to-end success rates, we also propose an intelligent backtracking
  mechanism that learns and recovers from its mistakes. Finally, we show that our
  ranking model can be trained on data from a generative auto-curriculum which
  samples representative goals from an LLM, runs the agent, and automatically
  evaluates it, with no manual annotation. Wilbur achieves state-of-the-art
  results on the WebVoyager benchmark, beating text-only models by 8% overall,
  and up to 36% on certain websites. On the same benchmark, Wilbur is within 5%
  of a strong multi-modal model despite only receiving textual inputs, and
  further analysis reveals a substantial number of failures are due to
  engineering challenges of operating the web.
paper_authors._id: 6618b3db144004cbc65962f7
paper_authors.name: Michael Lutz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-19T07:39:08.688Z
paper_authors._id: 61eba27eca2a7c5f718c7d50
paper_authors.avatarUrl: /avatars/b00ab396912ddcd8c7470dd82b575c53.svg
paper_authors.fullname: Michael Lutz
paper_authors.type: user
paper_authors.user: michaellutz
paper_authors._id: 6618b3db144004cbc65962f8
paper_authors.name: Arth Bohra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:20:13.994Z
paper_authors._id: 6369a9101e9d04886d595eef
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667868923415-noauth.jpeg
paper_authors.fullname: Arth Bohra
paper_authors.type: user
paper_authors.user: arthbohra
paper_authors._id: 6618b3db144004cbc65962f9
paper_authors.name: Manvel Saroyan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:20:22.106Z
paper_authors._id: 6619015111548294ccb6fa95
paper_authors.avatarUrl: /avatars/01f9c9bfb6361b009892ff49131a0646.svg
paper_authors.fullname: Manvel Saroyan
paper_authors.type: user
paper_authors.user: saroyanm
paper_authors._id: 6618b3db144004cbc65962fa
paper_authors.name: Artem Harutyunyan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T04:20:23.668Z
paper_authors._id: 6618b532e9f5bc959a8a9cb9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6618b532e9f5bc959a8a9cb9/--ZSoWIHaS9BWidPr5dGv.png
paper_authors.fullname: Artem H
paper_authors.type: user
paper_authors.user: hartem
paper_authors._id: 6618b3db144004cbc65962fb
paper_authors.name: Giovanni Campagna
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T04:32:34.205Z
paper_authors._id: 5feb674e9427fcf943fc19e6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5feb674e9427fcf943fc19e6/2gq_3nLJozZHItcpRxRuX.jpeg
paper_authors.fullname: Giovanni Campagna
paper_authors.type: user
paper_authors.user: gcampax
paper_title: WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents
paper_summary: In the realm of web agent research, achieving both generalization and
  accuracy remains a challenging problem. Due to high variance in website
  structure, existing approaches often fail. Moreover, existing fine-tuning and
  in-context learning techniques fail to generalize across multiple websites. We
  introduce Wilbur, an approach that uses a differentiable ranking model and a
  novel instruction synthesis technique to optimally populate a black-box large
  language model's prompt with task demonstrations from previous runs. To
  maximize end-to-end success rates, we also propose an intelligent backtracking
  mechanism that learns and recovers from its mistakes. Finally, we show that our
  ranking model can be trained on data from a generative auto-curriculum which
  samples representative goals from an LLM, runs the agent, and automatically
  evaluates it, with no manual annotation. Wilbur achieves state-of-the-art
  results on the WebVoyager benchmark, beating text-only models by 8% overall,
  and up to 36% on certain websites. On the same benchmark, Wilbur is within 5%
  of a strong multi-modal model despite only receiving textual inputs, and
  further analysis reveals a substantial number of failures are due to
  engineering challenges of operating the web.
paper_ai_summary: Wilbur uses a differentiable ranking model and instruction synthesis to fine-tune a large language model for web agent tasks, achieving state-of-the-art results with text-only inputs and superior performance on the WebVoyager benchmark.
paper_ai_keywords: differentiable ranking model instruction synthesis generative auto-curriculum large language model prompt population intelligent backtracking state-of-the-art WebVoyager benchmark multi-modal model