title: Robot Utility Models: General Policies for Zero-Shot Deployment in New
    Environments
summary: Robot models, particularly those trained with large amounts of data, have
  recently shown a plethora of real-world manipulation and navigation
  capabilities. Several independent efforts have shown that given sufficient
  training data in an environment, robot policies can generalize to demonstrated
  variations in that environment. However, needing to finetune robot models to
  every new environment stands in stark contrast to models in language or vision
  that can be deployed zero-shot for open-world problems. In this work, we
  present Robot Utility Models (RUMs), a framework for training and deploying
  zero-shot robot policies that can directly generalize to new environments
  without any finetuning. To create RUMs efficiently, we develop new tools to
  quickly collect data for mobile manipulation tasks, integrate such data into a
  policy with multi-modal imitation learning, and deploy policies on-device on
  Hello Robot Stretch, a cheap commodity robot, with an external mLLM verifier
  for retrying. We train five such utility models for opening cabinet doors,
  opening drawers, picking up napkins, picking up paper bags, and reorienting
  fallen objects. Our system, on average, achieves 90% success rate in unseen,
  novel environments interacting with unseen objects. Moreover, the utility
  models can also succeed in different robot and camera set-ups with no further
  data, training, or fine-tuning. Primary among our lessons are the importance of
  training data over training algorithm and policy class, guidance about data
  scaling, necessity for diverse yet high-quality demonstrations, and a recipe
  for robot introspection and retrying to improve performance on individual
  environments. Our code, data, models, hardware designs, as well as our
  experiment and deployment videos are open sourced and can be found on our
  project website: https://robotutilitymodels.com
paper_authors._id: 66e03fa39e29401825167fa7
paper_authors.name: Haritheja Etukuru
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-11T15:01:35.137Z
paper_authors._id: 65d16b116130ef7be0d212ef
paper_authors.avatarUrl: /avatars/5adda71719cc48f9a6f18ff67e279bb6.svg
paper_authors.fullname: Haritheja
paper_authors.type: user
paper_authors.user: haritheja
paper_authors._id: 66e03fa39e29401825167fa8
paper_authors.name: Norihito Naka
paper_authors._id: 66e03fa39e29401825167fa9
paper_authors.name: Zijin Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T15:40:14.804Z
paper_authors._id: 641e118d921e9c5d8481069b
paper_authors.avatarUrl: /avatars/8245951bf143e16a9447e9feccf5fef8.svg
paper_authors.fullname: Zijin Hu
paper_authors.type: user
paper_authors.user: Z1j1n
paper_authors._id: 66e03fa39e29401825167faa
paper_authors.name: Seungjae Lee
paper_authors._id: 66e03fa39e29401825167fab
paper_authors.name: Julian Mehu
paper_authors._id: 66e03fa39e29401825167fac
paper_authors.name: Aaron Edsinger
paper_authors._id: 66e03fa39e29401825167fad
paper_authors.name: Chris Paxton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T15:39:48.313Z
paper_authors._id: 62b13b28b7053ab38d2d5f96
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b13b28b7053ab38d2d5f96/V6780Omima7sNW5E4TEeW.png
paper_authors.fullname: Chris Paxton
paper_authors.type: user
paper_authors.user: cpaxton
paper_authors._id: 66e03fa39e29401825167fae
paper_authors.name: Soumith Chintala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T15:39:42.300Z
paper_authors._id: 64b81f399ebb69a79ffec095
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/2nSfGQk0O72prUr3GfV4V.png
paper_authors.fullname: Soumith Chintala
paper_authors.type: user
paper_authors.user: soumithchintala
paper_authors._id: 66e03fa39e29401825167faf
paper_authors.name: Lerrel Pinto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T15:39:54.148Z
paper_authors._id: 6388cdb6685efaedd15dc0f3
paper_authors.avatarUrl: /avatars/b2b74454bf86bf61fd90ba088644f547.svg
paper_authors.fullname: Lerrel Pinto
paper_authors.type: user
paper_authors.user: lerrel
paper_authors._id: 66e03fa39e29401825167fb0
paper_authors.name: Nur Muhammad Mahi Shafiullah
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T13:03:13.122Z
paper_authors._id: 630e567f8df86f1e5bf0d837
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630e567f8df86f1e5bf0d837/Mje-2ahgmMET2jzqLafhy.jpeg
paper_authors.fullname: Mahi Shafiullah
paper_authors.type: user
paper_authors.user: notmahi
paper_title: Robot Utility Models: General Policies for Zero-Shot Deployment in New
    Environments
paper_summary: Robot models, particularly those trained with large amounts of data, have
  recently shown a plethora of real-world manipulation and navigation
  capabilities. Several independent efforts have shown that given sufficient
  training data in an environment, robot policies can generalize to demonstrated
  variations in that environment. However, needing to finetune robot models to
  every new environment stands in stark contrast to models in language or vision
  that can be deployed zero-shot for open-world problems. In this work, we
  present Robot Utility Models (RUMs), a framework for training and deploying
  zero-shot robot policies that can directly generalize to new environments
  without any finetuning. To create RUMs efficiently, we develop new tools to
  quickly collect data for mobile manipulation tasks, integrate such data into a
  policy with multi-modal imitation learning, and deploy policies on-device on
  Hello Robot Stretch, a cheap commodity robot, with an external mLLM verifier
  for retrying. We train five such utility models for opening cabinet doors,
  opening drawers, picking up napkins, picking up paper bags, and reorienting
  fallen objects. Our system, on average, achieves 90% success rate in unseen,
  novel environments interacting with unseen objects. Moreover, the utility
  models can also succeed in different robot and camera set-ups with no further
  data, training, or fine-tuning. Primary among our lessons are the importance of
  training data over training algorithm and policy class, guidance about data
  scaling, necessity for diverse yet high-quality demonstrations, and a recipe
  for robot introspection and retrying to improve performance on individual
  environments. Our code, data, models, hardware designs, as well as our
  experiment and deployment videos are open sourced and can be found on our
  project website: https://robotutilitymodels.com
paper_ai_keywords: multi-modal imitation learning on-device deployment external mLLM verifier retrying utility models success rate robot introspection