Embodied-RAG: General non-parametric Embodied Memory for Retrieval and
  Generation
There is no limit to how much a robot might explore and learn, but all of
that knowledge needs to be searchable and actionable. Within language research,
retrieval augmented generation (RAG) has become the workhouse of large-scale
non-parametric knowledge, however existing techniques do not directly transfer
to the embodied domain, which is multimodal, data is highly correlated, and
perception requires abstraction.
  To address these challenges, we introduce Embodied-RAG, a framework that
enhances the foundational model of an embodied agent with a non-parametric
memory system capable of autonomously constructing hierarchical knowledge for
both navigation and language generation. Embodied-RAG handles a full range of
spatial and semantic resolutions across diverse environments and query types,
whether for a specific object or a holistic description of ambiance. At its
core, Embodied-RAG's memory is structured as a semantic forest, storing
language descriptions at varying levels of detail. This hierarchical
organization allows the system to efficiently generate context-sensitive
outputs across different robotic platforms. We demonstrate that Embodied-RAG
effectively bridges RAG to the robotics domain, successfully handling over 200
explanation and navigation queries across 19 environments, highlighting its
promise for general-purpose non-parametric system for embodied agents.
66fd74e7244c47597e953644
Quanting Xie
claimed_verified
2024-10-03T08:31:37.659Z
66fc728cf553900c024a4130
/avatars/dda2c9a3ce048020ea86d45a29f20b41.svg
Xie
user
Quanting
66fd74e7244c47597e953645
So Yeon Min
66fd74e7244c47597e953646
Tianyi Zhang
66fd74e7244c47597e953647
Aarav Bajaj
66fd74e7244c47597e953648
Ruslan Salakhutdinov
66fd74e7244c47597e953649
Matthew Johnson-Roberson
66fd74e7244c47597e95364a
Yonatan Bisk
Embodied-RAG: General non-parametric Embodied Memory for Retrieval and
  Generation
There is no limit to how much a robot might explore and learn, but all of
that knowledge needs to be searchable and actionable. Within language research,
retrieval augmented generation (RAG) has become the workhouse of large-scale
non-parametric knowledge, however existing techniques do not directly transfer
to the embodied domain, which is multimodal, data is highly correlated, and
perception requires abstraction.
  To address these challenges, we introduce Embodied-RAG, a framework that
enhances the foundational model of an embodied agent with a non-parametric
memory system capable of autonomously constructing hierarchical knowledge for
both navigation and language generation. Embodied-RAG handles a full range of
spatial and semantic resolutions across diverse environments and query types,
whether for a specific object or a holistic description of ambiance. At its
core, Embodied-RAG's memory is structured as a semantic forest, storing
language descriptions at varying levels of detail. This hierarchical
organization allows the system to efficiently generate context-sensitive
outputs across different robotic platforms. We demonstrate that Embodied-RAG
effectively bridges RAG to the robotics domain, successfully handling over 200
explanation and navigation queries across 19 environments, highlighting its
promise for general-purpose non-parametric system for embodied agents.
Embodied-RAG, a framework for embodied agents, combines non-parametric memory with hierarchical spatial and semantic knowledge, enabling effective retrieval and generation in diverse robotic environments.
Embodied-RAG retrieval augmented generation (RAG) non-parametric knowledge embodied agent hierarchical knowledge spatial resolutions semantic resolutions semantic forest context-sensitive outputs