title: Embodied-RAG: General non-parametric Embodied Memory for Retrieval and
    Generation
summary: There is no limit to how much a robot might explore and learn, but all of
  that knowledge needs to be searchable and actionable. Within language research,
  retrieval augmented generation (RAG) has become the workhouse of large-scale
  non-parametric knowledge, however existing techniques do not directly transfer
  to the embodied domain, which is multimodal, data is highly correlated, and
  perception requires abstraction.
    To address these challenges, we introduce Embodied-RAG, a framework that
  enhances the foundational model of an embodied agent with a non-parametric
  memory system capable of autonomously constructing hierarchical knowledge for
  both navigation and language generation. Embodied-RAG handles a full range of
  spatial and semantic resolutions across diverse environments and query types,
  whether for a specific object or a holistic description of ambiance. At its
  core, Embodied-RAG's memory is structured as a semantic forest, storing
  language descriptions at varying levels of detail. This hierarchical
  organization allows the system to efficiently generate context-sensitive
  outputs across different robotic platforms. We demonstrate that Embodied-RAG
  effectively bridges RAG to the robotics domain, successfully handling over 200
  explanation and navigation queries across 19 environments, highlighting its
  promise for general-purpose non-parametric system for embodied agents.
paper_authors._id: 66fd74e7244c47597e953644
paper_authors.name: Quanting Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-03T08:31:37.659Z
paper_authors._id: 66fc728cf553900c024a4130
paper_authors.avatarUrl: /avatars/dda2c9a3ce048020ea86d45a29f20b41.svg
paper_authors.fullname: Xie
paper_authors.type: user
paper_authors.user: Quanting
paper_authors._id: 66fd74e7244c47597e953645
paper_authors.name: So Yeon Min
paper_authors._id: 66fd74e7244c47597e953646
paper_authors.name: Tianyi Zhang
paper_authors._id: 66fd74e7244c47597e953647
paper_authors.name: Aarav Bajaj
paper_authors._id: 66fd74e7244c47597e953648
paper_authors.name: Ruslan Salakhutdinov
paper_authors._id: 66fd74e7244c47597e953649
paper_authors.name: Matthew Johnson-Roberson
paper_authors._id: 66fd74e7244c47597e95364a
paper_authors.name: Yonatan Bisk
paper_title: Embodied-RAG: General non-parametric Embodied Memory for Retrieval and
    Generation
paper_summary: There is no limit to how much a robot might explore and learn, but all of
  that knowledge needs to be searchable and actionable. Within language research,
  retrieval augmented generation (RAG) has become the workhouse of large-scale
  non-parametric knowledge, however existing techniques do not directly transfer
  to the embodied domain, which is multimodal, data is highly correlated, and
  perception requires abstraction.
    To address these challenges, we introduce Embodied-RAG, a framework that
  enhances the foundational model of an embodied agent with a non-parametric
  memory system capable of autonomously constructing hierarchical knowledge for
  both navigation and language generation. Embodied-RAG handles a full range of
  spatial and semantic resolutions across diverse environments and query types,
  whether for a specific object or a holistic description of ambiance. At its
  core, Embodied-RAG's memory is structured as a semantic forest, storing
  language descriptions at varying levels of detail. This hierarchical
  organization allows the system to efficiently generate context-sensitive
  outputs across different robotic platforms. We demonstrate that Embodied-RAG
  effectively bridges RAG to the robotics domain, successfully handling over 200
  explanation and navigation queries across 19 environments, highlighting its
  promise for general-purpose non-parametric system for embodied agents.
paper_ai_keywords: retrieval augmented generation (RAG) Embodied-RAG non-parametric memory system hierarchical knowledge navigation language generation semantic forest context-sensitive outputs general-purpose non-parametric system embodied agents