title: DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile
    Manipulation
summary: Significant progress has been made in open-vocabulary mobile manipulation,
  where the goal is for a robot to perform tasks in any environment given a
  natural language description. However, most current systems assume a static
  environment, which limits the system's applicability in real-world scenarios
  where environments frequently change due to human intervention or the robot's
  own actions. In this work, we present DynaMem, a new approach to open-world
  mobile manipulation that uses a dynamic spatio-semantic memory to represent a
  robot's environment. DynaMem constructs a 3D data structure to maintain a
  dynamic memory of point clouds, and answers open-vocabulary object localization
  queries using multimodal LLMs or open-vocabulary features generated by
  state-of-the-art vision-language models. Powered by DynaMem, our robots can
  explore novel environments, search for objects not found in memory, and
  continuously update the memory as objects move, appear, or disappear in the
  scene. We run extensive experiments on the Stretch SE3 robots in three real and
  nine offline scenes, and achieve an average pick-and-drop success rate of 70%
  on non-stationary objects, which is more than a 2x improvement over
  state-of-the-art static systems. Our code as well as our experiment and
  deployment videos are open sourced and can be found on our project website:
  https://dynamem.github.io/
paper_authors._id: 672d85312ed6d70d4225f9d2
paper_authors.name: Peiqi Liu
paper_authors._id: 672d85312ed6d70d4225f9d3
paper_authors.name: Zhanqiu Guo
paper_authors._id: 672d85312ed6d70d4225f9d4
paper_authors.name: Mohit Warke
paper_authors._id: 672d85312ed6d70d4225f9d5
paper_authors.name: Soumith Chintala
paper_authors._id: 672d85312ed6d70d4225f9d6
paper_authors.name: Chris Paxton
paper_authors._id: 672d85312ed6d70d4225f9d7
paper_authors.name: Nur Muhammad Mahi Shafiullah
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:31:22.728Z
paper_authors._id: 630e567f8df86f1e5bf0d837
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630e567f8df86f1e5bf0d837/Mje-2ahgmMET2jzqLafhy.jpeg
paper_authors.fullname: Mahi Shafiullah
paper_authors.type: user
paper_authors.user: notmahi
paper_authors._id: 672d85312ed6d70d4225f9d8
paper_authors.name: Lerrel Pinto
paper_title: DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile
    Manipulation
paper_summary: Significant progress has been made in open-vocabulary mobile manipulation,
  where the goal is for a robot to perform tasks in any environment given a
  natural language description. However, most current systems assume a static
  environment, which limits the system's applicability in real-world scenarios
  where environments frequently change due to human intervention or the robot's
  own actions. In this work, we present DynaMem, a new approach to open-world
  mobile manipulation that uses a dynamic spatio-semantic memory to represent a
  robot's environment. DynaMem constructs a 3D data structure to maintain a
  dynamic memory of point clouds, and answers open-vocabulary object localization
  queries using multimodal LLMs or open-vocabulary features generated by
  state-of-the-art vision-language models. Powered by DynaMem, our robots can
  explore novel environments, search for objects not found in memory, and
  continuously update the memory as objects move, appear, or disappear in the
  scene. We run extensive experiments on the Stretch SE3 robots in three real and
  nine offline scenes, and achieve an average pick-and-drop success rate of 70%
  on non-stationary objects, which is more than a 2x improvement over
  state-of-the-art static systems. Our code as well as our experiment and
  deployment videos are open sourced and can be found on our project website:
  https://dynamem.github.io/
paper_ai_keywords: DynaMem Visual Question memory 3D data structure point clouds open-vocabulary object localization multimodal LLMs state-of-the-art vision-language models dynamic spatio-semantic memory pick-and-drop success rate non-stationary objects