Robot Utility Models: General Policies for Zero-Shot Deployment in New
  Environments
Robot models, particularly those trained with large amounts of data, have
recently shown a plethora of real-world manipulation and navigation
capabilities. Several independent efforts have shown that given sufficient
training data in an environment, robot policies can generalize to demonstrated
variations in that environment. However, needing to finetune robot models to
every new environment stands in stark contrast to models in language or vision
that can be deployed zero-shot for open-world problems. In this work, we
present Robot Utility Models (RUMs), a framework for training and deploying
zero-shot robot policies that can directly generalize to new environments
without any finetuning. To create RUMs efficiently, we develop new tools to
quickly collect data for mobile manipulation tasks, integrate such data into a
policy with multi-modal imitation learning, and deploy policies on-device on
Hello Robot Stretch, a cheap commodity robot, with an external mLLM verifier
for retrying. We train five such utility models for opening cabinet doors,
opening drawers, picking up napkins, picking up paper bags, and reorienting
fallen objects. Our system, on average, achieves 90% success rate in unseen,
novel environments interacting with unseen objects. Moreover, the utility
models can also succeed in different robot and camera set-ups with no further
data, training, or fine-tuning. Primary among our lessons are the importance of
training data over training algorithm and policy class, guidance about data
scaling, necessity for diverse yet high-quality demonstrations, and a recipe
for robot introspection and retrying to improve performance on individual
environments. Our code, data, models, hardware designs, as well as our
experiment and deployment videos are open sourced and can be found on our
project website: https://robotutilitymodels.com
66e03fa39e29401825167fa7
Haritheja Etukuru
claimed_verified
2024-09-11T15:01:35.137Z
65d16b116130ef7be0d212ef
/avatars/5adda71719cc48f9a6f18ff67e279bb6.svg
Haritheja
user
haritheja
66e03fa39e29401825167fa8
Norihito Naka
66e03fa39e29401825167fa9
Zijin Hu
admin_assigned
2024-09-10T15:40:14.804Z
641e118d921e9c5d8481069b
/avatars/8245951bf143e16a9447e9feccf5fef8.svg
Zijin Hu
user
Z1j1n
66e03fa39e29401825167faa
Seungjae Lee
66e03fa39e29401825167fab
Julian Mehu
66e03fa39e29401825167fac
Aaron Edsinger
66e03fa39e29401825167fad
Chris Paxton
admin_assigned
2024-09-10T15:39:48.313Z
62b13b28b7053ab38d2d5f96
https://cdn-avatars.huggingface.co/v1/production/uploads/62b13b28b7053ab38d2d5f96/V6780Omima7sNW5E4TEeW.png
Chris Paxton
user
cpaxton
66e03fa39e29401825167fae
Soumith Chintala
admin_assigned
2024-09-10T15:39:42.300Z
64b81f399ebb69a79ffec095
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/2nSfGQk0O72prUr3GfV4V.png
Soumith Chintala
user
soumithchintala
66e03fa39e29401825167faf
Lerrel Pinto
admin_assigned
2024-09-10T15:39:54.148Z
6388cdb6685efaedd15dc0f3
/avatars/b2b74454bf86bf61fd90ba088644f547.svg
Lerrel Pinto
user
lerrel
66e03fa39e29401825167fb0
Nur Muhammad Mahi Shafiullah
claimed_verified
2024-09-10T13:03:13.122Z
630e567f8df86f1e5bf0d837
https://cdn-avatars.huggingface.co/v1/production/uploads/630e567f8df86f1e5bf0d837/Mje-2ahgmMET2jzqLafhy.jpeg
Mahi Shafiullah
user
notmahi
Robot Utility Models: General Policies for Zero-Shot Deployment in New
  Environments
Robot models, particularly those trained with large amounts of data, have
recently shown a plethora of real-world manipulation and navigation
capabilities. Several independent efforts have shown that given sufficient
training data in an environment, robot policies can generalize to demonstrated
variations in that environment. However, needing to finetune robot models to
every new environment stands in stark contrast to models in language or vision
that can be deployed zero-shot for open-world problems. In this work, we
present Robot Utility Models (RUMs), a framework for training and deploying
zero-shot robot policies that can directly generalize to new environments
without any finetuning. To create RUMs efficiently, we develop new tools to
quickly collect data for mobile manipulation tasks, integrate such data into a
policy with multi-modal imitation learning, and deploy policies on-device on
Hello Robot Stretch, a cheap commodity robot, with an external mLLM verifier
for retrying. We train five such utility models for opening cabinet doors,
opening drawers, picking up napkins, picking up paper bags, and reorienting
fallen objects. Our system, on average, achieves 90% success rate in unseen,
novel environments interacting with unseen objects. Moreover, the utility
models can also succeed in different robot and camera set-ups with no further
data, training, or fine-tuning. Primary among our lessons are the importance of
training data over training algorithm and policy class, guidance about data
scaling, necessity for diverse yet high-quality demonstrations, and a recipe
for robot introspection and retrying to improve performance on individual
environments. Our code, data, models, hardware designs, as well as our
experiment and deployment videos are open sourced and can be found on our
project website: https://robotutilitymodels.com
multi-modal imitation learning on-device deployment external mLLM verifier retrying utility models success rate robot introspection