AutoCrawler: A Progressive Understanding Web Agent for Web Crawler
  Generation
Web automation is a significant technique that accomplishes complicated web
tasks by automating common web actions, enhancing operational efficiency, and
reducing the need for manual intervention. Traditional methods, such as
wrappers, suffer from limited adaptability and scalability when faced with a
new website. On the other hand, generative agents empowered by large language
models (LLMs) exhibit poor performance and reusability in open-world scenarios.
In this work, we introduce a crawler generation task for vertical information
web pages and the paradigm of combining LLMs with crawlers, which helps
crawlers handle diverse and changing web environments more efficiently. We
propose AutoCrawler, a two-stage framework that leverages the hierarchical
structure of HTML for progressive understanding. Through top-down and step-back
operations, AutoCrawler can learn from erroneous actions and continuously prune
HTML for better action generation. We conduct comprehensive experiments with
multiple LLMs and demonstrate the effectiveness of our framework. Resources of
this paper can be found at https://github.com/EZ-hwh/AutoCrawler
6625f2f622668df9ad4f88be
Wenhao Huang
admin_assigned
2024-04-22T14:04:42.591Z
641e5bf65f274a0a92c2f6a2
/avatars/c15a54c51998c0e6367685e8e1737ec9.svg
Wenhao Huang
user
EZ-hwh
6625f2f622668df9ad4f88bf
Chenghao Peng
admin_assigned
2024-04-22T14:04:49.324Z
64ae014d3b0219010e38c48d
/avatars/e54bd19f51b544f23e48a9d1a9d8e000.svg
chenghaopeng
user
chenghaopeng
6625f2f622668df9ad4f88c0
Zhixu Li
admin_assigned
2024-04-22T14:04:55.400Z
661dd71d2ae9013218415e6f
/avatars/389883e5886628c07cb0b08fc8c93c3b.svg
Zhixu Li
user
ZhixuLi
6625f2f622668df9ad4f88c1
Jiaqing Liang
admin_assigned
2024-04-22T14:05:07.110Z
634014022c72fd4ca251fed7
/avatars/32a85d41f7435f7facfd580d9d124c33.svg
liangjiaqing
user
liangjiaqing
6625f2f622668df9ad4f88c2
Yanghua Xiao
6625f2f622668df9ad4f88c3
Liqian Wen
6625f2f622668df9ad4f88c4
Zulong Chen
admin_assigned
2024-04-22T14:05:45.240Z
62a704d6c1d74b7cbccf930b
https://cdn-avatars.huggingface.co/v1/production/uploads/62a704d6c1d74b7cbccf930b/cK0CgMsQPIb3oIQvOuh-G.png
chen
user
Zulong
AutoCrawler: A Progressive Understanding Web Agent for Web Crawler
  Generation
Web automation is a significant technique that accomplishes complicated web
tasks by automating common web actions, enhancing operational efficiency, and
reducing the need for manual intervention. Traditional methods, such as
wrappers, suffer from limited adaptability and scalability when faced with a
new website. On the other hand, generative agents empowered by large language
models (LLMs) exhibit poor performance and reusability in open-world scenarios.
In this work, we introduce a crawler generation task for vertical information
web pages and the paradigm of combining LLMs with crawlers, which helps
crawlers handle diverse and changing web environments more efficiently. We
propose AutoCrawler, a two-stage framework that leverages the hierarchical
structure of HTML for progressive understanding. Through top-down and step-back
operations, AutoCrawler can learn from erroneous actions and continuously prune
HTML for better action generation. We conduct comprehensive experiments with
multiple LLMs and demonstrate the effectiveness of our framework. Resources of
this paper can be found at https://github.com/EZ-hwh/AutoCrawler
AutoCrawler combines LLMs with crawlers to efficiently handle diverse and changing web environments using a hierarchical HTML structure for progressive learning.
LLMs crawlers vertical information web pages hierarchical structure HTML top-down operations step-back operations action generation