Embodied Web Agents: Bridging Physical-Digital Realms for Integrated
  Agent Intelligence
AI agents today are mostly siloed - they either retrieve and reason over vast
amount of digital information and knowledge obtained online; or interact with
the physical world through embodied perception, planning and action - but
rarely both. This separation limits their ability to solve tasks that require
integrated physical and digital intelligence, such as cooking from online
recipes, navigating with dynamic map data, or interpreting real-world landmarks
using web knowledge. We introduce Embodied Web Agents, a novel paradigm for AI
agents that fluidly bridge embodiment and web-scale reasoning. To
operationalize this concept, we first develop the Embodied Web Agents task
environments, a unified simulation platform that tightly integrates realistic
3D indoor and outdoor environments with functional web interfaces. Building
upon this platform, we construct and release the Embodied Web Agents Benchmark,
which encompasses a diverse suite of tasks including cooking, navigation,
shopping, tourism, and geolocation - all requiring coordinated reasoning across
physical and digital realms for systematic assessment of cross-domain
intelligence. Experimental results reveal significant performance gaps between
state-of-the-art AI systems and human capabilities, establishing both
challenges and opportunities at the intersection of embodied cognition and
web-scale knowledge access. All datasets, codes and websites are publicly
available at our project page https://embodied-web-agent.github.io/.
68536b2399bf39f9665c794c
Yining Hong
68536b2399bf39f9665c794d
Rui Sun
68536b2399bf39f9665c794e
Bingxuan Li
68536b2399bf39f9665c794f
Xingcheng Yao
68536b2399bf39f9665c7950
Maxine Wu
68536b2399bf39f9665c7951
Alexander Chien
68536b2399bf39f9665c7952
Da Yin
68536b2399bf39f9665c7953
Ying Nian Wu
68536b2399bf39f9665c7954
Zhecan James Wang
68536b2399bf39f9665c7955
Kai-Wei Chang
Embodied Web Agents: Bridging Physical-Digital Realms for Integrated
  Agent Intelligence
AI agents today are mostly siloed - they either retrieve and reason over vast
amount of digital information and knowledge obtained online; or interact with
the physical world through embodied perception, planning and action - but
rarely both. This separation limits their ability to solve tasks that require
integrated physical and digital intelligence, such as cooking from online
recipes, navigating with dynamic map data, or interpreting real-world landmarks
using web knowledge. We introduce Embodied Web Agents, a novel paradigm for AI
agents that fluidly bridge embodiment and web-scale reasoning. To
operationalize this concept, we first develop the Embodied Web Agents task
environments, a unified simulation platform that tightly integrates realistic
3D indoor and outdoor environments with functional web interfaces. Building
upon this platform, we construct and release the Embodied Web Agents Benchmark,
which encompasses a diverse suite of tasks including cooking, navigation,
shopping, tourism, and geolocation - all requiring coordinated reasoning across
physical and digital realms for systematic assessment of cross-domain
intelligence. Experimental results reveal significant performance gaps between
state-of-the-art AI systems and human capabilities, establishing both
challenges and opportunities at the intersection of embodied cognition and
web-scale knowledge access. All datasets, codes and websites are publicly
available at our project page https://embodied-web-agent.github.io/.
Embodied Web Agents integrate physical interaction and web-scale reasoning to assess cross-domain intelligence in a novel benchmark environment.
Embodied Web Agents task environments simulation platform 3D indoor and outdoor environments functional web interfaces Embodied Web Agents Benchmark systematic assessment cross-domain intelligence embodied cognition