title: AutoCrawler: A Progressive Understanding Web Agent for Web Crawler
    Generation
summary: Web automation is a significant technique that accomplishes complicated web
  tasks by automating common web actions, enhancing operational efficiency, and
  reducing the need for manual intervention. Traditional methods, such as
  wrappers, suffer from limited adaptability and scalability when faced with a
  new website. On the other hand, generative agents empowered by large language
  models (LLMs) exhibit poor performance and reusability in open-world scenarios.
  In this work, we introduce a crawler generation task for vertical information
  web pages and the paradigm of combining LLMs with crawlers, which helps
  crawlers handle diverse and changing web environments more efficiently. We
  propose AutoCrawler, a two-stage framework that leverages the hierarchical
  structure of HTML for progressive understanding. Through top-down and step-back
  operations, AutoCrawler can learn from erroneous actions and continuously prune
  HTML for better action generation. We conduct comprehensive experiments with
  multiple LLMs and demonstrate the effectiveness of our framework. Resources of
  this paper can be found at https://github.com/EZ-hwh/AutoCrawler
paper_authors._id: 6625f2f622668df9ad4f88be
paper_authors.name: Wenhao Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:04:42.591Z
paper_authors._id: 641e5bf65f274a0a92c2f6a2
paper_authors.avatarUrl: /avatars/c15a54c51998c0e6367685e8e1737ec9.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: EZ-hwh
paper_authors._id: 6625f2f622668df9ad4f88bf
paper_authors.name: Chenghao Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:04:49.324Z
paper_authors._id: 64ae014d3b0219010e38c48d
paper_authors.avatarUrl: /avatars/e54bd19f51b544f23e48a9d1a9d8e000.svg
paper_authors.fullname: chenghaopeng
paper_authors.type: user
paper_authors.user: chenghaopeng
paper_authors._id: 6625f2f622668df9ad4f88c0
paper_authors.name: Zhixu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:04:55.400Z
paper_authors._id: 661dd71d2ae9013218415e6f
paper_authors.avatarUrl: /avatars/389883e5886628c07cb0b08fc8c93c3b.svg
paper_authors.fullname: Zhixu Li
paper_authors.type: user
paper_authors.user: ZhixuLi
paper_authors._id: 6625f2f622668df9ad4f88c1
paper_authors.name: Jiaqing Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:05:07.110Z
paper_authors._id: 634014022c72fd4ca251fed7
paper_authors.avatarUrl: /avatars/32a85d41f7435f7facfd580d9d124c33.svg
paper_authors.fullname: liangjiaqing
paper_authors.type: user
paper_authors.user: liangjiaqing
paper_authors._id: 6625f2f622668df9ad4f88c2
paper_authors.name: Yanghua Xiao
paper_authors._id: 6625f2f622668df9ad4f88c3
paper_authors.name: Liqian Wen
paper_authors._id: 6625f2f622668df9ad4f88c4
paper_authors.name: Zulong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:05:45.240Z
paper_authors._id: 62a704d6c1d74b7cbccf930b
paper_authors.avatarUrl: /avatars/ed7e136d5d80bfaa7d6731b9ad4b622f.svg
paper_authors.fullname: chen
paper_authors.type: user
paper_authors.user: Zulong
paper_title: AutoCrawler: A Progressive Understanding Web Agent for Web Crawler
    Generation
paper_summary: Web automation is a significant technique that accomplishes complicated web
  tasks by automating common web actions, enhancing operational efficiency, and
  reducing the need for manual intervention. Traditional methods, such as
  wrappers, suffer from limited adaptability and scalability when faced with a
  new website. On the other hand, generative agents empowered by large language
  models (LLMs) exhibit poor performance and reusability in open-world scenarios.
  In this work, we introduce a crawler generation task for vertical information
  web pages and the paradigm of combining LLMs with crawlers, which helps
  crawlers handle diverse and changing web environments more efficiently. We
  propose AutoCrawler, a two-stage framework that leverages the hierarchical
  structure of HTML for progressive understanding. Through top-down and step-back
  operations, AutoCrawler can learn from erroneous actions and continuously prune
  HTML for better action generation. We conduct comprehensive experiments with
  multiple LLMs and demonstrate the effectiveness of our framework. Resources of
  this paper can be found at https://github.com/EZ-hwh/AutoCrawler
paper_ai_keywords: generative agents large language models (LLMs) crawler generation task vertical information hierarchical structure of HTML top-down step-back operations AutoCrawler two-stage framework action generation erroneous actions continuous pruning end-to-end understanding