DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile
  Manipulation
Significant progress has been made in open-vocabulary mobile manipulation,
where the goal is for a robot to perform tasks in any environment given a
natural language description. However, most current systems assume a static
environment, which limits the system's applicability in real-world scenarios
where environments frequently change due to human intervention or the robot's
own actions. In this work, we present DynaMem, a new approach to open-world
mobile manipulation that uses a dynamic spatio-semantic memory to represent a
robot's environment. DynaMem constructs a 3D data structure to maintain a
dynamic memory of point clouds, and answers open-vocabulary object localization
queries using multimodal LLMs or open-vocabulary features generated by
state-of-the-art vision-language models. Powered by DynaMem, our robots can
explore novel environments, search for objects not found in memory, and
continuously update the memory as objects move, appear, or disappear in the
scene. We run extensive experiments on the Stretch SE3 robots in three real and
nine offline scenes, and achieve an average pick-and-drop success rate of 70%
on non-stationary objects, which is more than a 2x improvement over
state-of-the-art static systems. Our code as well as our experiment and
deployment videos are open sourced and can be found on our project website:
https://dynamem.github.io/
672d85312ed6d70d4225f9d2
Peiqi Liu
672d85312ed6d70d4225f9d3
Zhanqiu Guo
672d85312ed6d70d4225f9d4
Mohit Warke
672d85312ed6d70d4225f9d5
Soumith Chintala
672d85312ed6d70d4225f9d6
Chris Paxton
672d85312ed6d70d4225f9d7
Nur Muhammad Mahi Shafiullah
claimed_verified
2024-11-15T09:31:22.728Z
630e567f8df86f1e5bf0d837
https://cdn-avatars.huggingface.co/v1/production/uploads/630e567f8df86f1e5bf0d837/Mje-2ahgmMET2jzqLafhy.jpeg
Mahi Shafiullah
user
notmahi
672d85312ed6d70d4225f9d8
Lerrel Pinto
DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile
  Manipulation
Significant progress has been made in open-vocabulary mobile manipulation,
where the goal is for a robot to perform tasks in any environment given a
natural language description. However, most current systems assume a static
environment, which limits the system's applicability in real-world scenarios
where environments frequently change due to human intervention or the robot's
own actions. In this work, we present DynaMem, a new approach to open-world
mobile manipulation that uses a dynamic spatio-semantic memory to represent a
robot's environment. DynaMem constructs a 3D data structure to maintain a
dynamic memory of point clouds, and answers open-vocabulary object localization
queries using multimodal LLMs or open-vocabulary features generated by
state-of-the-art vision-language models. Powered by DynaMem, our robots can
explore novel environments, search for objects not found in memory, and
continuously update the memory as objects move, appear, or disappear in the
scene. We run extensive experiments on the Stretch SE3 robots in three real and
nine offline scenes, and achieve an average pick-and-drop success rate of 70%
on non-stationary objects, which is more than a 2x improvement over
state-of-the-art static systems. Our code as well as our experiment and
deployment videos are open sourced and can be found on our project website:
https://dynamem.github.io/
DynaMem, a dynamic memory system for robots, enhances open-vocabulary mobile manipulation by representing and updating 3D environments, achieving high success rates in non-stationary tasks.
dynamic spatio-semantic memory 3D data structure point clouds multimodal LLMs open-vocabulary features vision-language models open-vocabulary object localization Stretch SE3 robots