title: HomeRobot: Open-Vocabulary Mobile Manipulation
summary: HomeRobot (noun): An affordable compliant robot that navigates homes and
  manipulates a wide range of objects in order to complete everyday tasks.
  Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object
  in any unseen environment, and placing it in a commanded location. This is a
  foundational challenge for robots to be useful assistants in human
  environments, because it involves tackling sub-problems from across robotics:
  perception, language understanding, navigation, and manipulation are all
  essential to OVMM. In addition, integration of the solutions to these
  sub-problems poses its own substantial challenges. To drive research in this
  area, we introduce the HomeRobot OVMM benchmark, where an agent navigates
  household environments to grasp novel objects and place them on target
  receptacles. HomeRobot has two components: a simulation component, which uses a
  large and diverse curated object set in new, high-quality multi-room home
  environments; and a real-world component, providing a software stack for the
  low-cost Hello Robot Stretch to encourage replication of real-world experiments
  across labs. We implement both reinforcement learning and heuristic
  (model-based) baselines and show evidence of sim-to-real transfer. Our
  baselines achieve a 20% success rate in the real world; our experiments
  identify ways future research work improve performance. See videos on our
  website: https://ovmm.github.io/.
paper_authors._id: 64927a832aa84c46de6ba8ea
paper_authors.name: Sriram Yenamandra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-21T09:16:41.054Z
paper_authors._id: 640d3ece4b5450a208b275fd
paper_authors.avatarUrl: /avatars/669e161448c0895b856065646d68c24c.svg
paper_authors.fullname: sriram
paper_authors.type: user
paper_authors.user: yvsriram
paper_authors._id: 64927a832aa84c46de6ba8eb
paper_authors.name: Arun Ramachandran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:33:11.751Z
paper_authors._id: 63ed2501f765928ceead06cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ed2501f765928ceead06cb/UffowP9cBW5CZoLo0BhP5.jpeg
paper_authors.fullname: Arun Ramachandran
paper_authors.type: user
paper_authors.user: JHurricane96
paper_authors._id: 64927a832aa84c46de6ba8ec
paper_authors.name: Karmesh Yadav
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:33:02.356Z
paper_authors._id: 63ed4eac679c2cc40ab86838
paper_authors.avatarUrl: /avatars/f9a0d4af63dcc713e77c3533d1809699.svg
paper_authors.fullname: Karmesh Yadav
paper_authors.type: user
paper_authors.user: ykarmesh
paper_authors._id: 64927a832aa84c46de6ba8ed
paper_authors.name: Austin Wang
paper_authors._id: 64927a832aa84c46de6ba8ee
paper_authors.name: Mukul Khanna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:34:00.351Z
paper_authors._id: 6387a0328b3c55fea3e4a13d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6387a0328b3c55fea3e4a13d/uxMoiqxD0GLKV95ItECJr.jpeg
paper_authors.fullname: Mukul Khanna
paper_authors.type: user
paper_authors.user: mukulkhanna
paper_authors._id: 64927a832aa84c46de6ba8ef
paper_authors.name: Theophile Gervet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:34:20.651Z
paper_authors._id: 644e822cd6001776ed7700e4
paper_authors.avatarUrl: /avatars/58a72e6bf8ede3cee3de54873231fd3e.svg
paper_authors.fullname: Theophile Gervet
paper_authors.type: user
paper_authors.user: TheophileGervet
paper_authors._id: 64927a832aa84c46de6ba8f0
paper_authors.name: Tsung-Yen Yang
paper_authors._id: 64927a832aa84c46de6ba8f1
paper_authors.name: Vidhi Jain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:35:03.268Z
paper_authors._id: 6335dd2717cccb7421bb0def
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6335dd2717cccb7421bb0def/yFjtuIjyyh6MY42pKLLjw.png
paper_authors.fullname: Vidhi Jain
paper_authors.type: user
paper_authors.user: vidhij
paper_authors._id: 64927a832aa84c46de6ba8f2
paper_authors.name: Alexander William Clegg
paper_authors._id: 64927a832aa84c46de6ba8f3
paper_authors.name: John Turner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:35:59.683Z
paper_authors._id: 63ee68535f1300034dde7848
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676570521120-63ee68535f1300034dde7848.jpeg
paper_authors.fullname: John Turner
paper_authors.type: user
paper_authors.user: JohnTurner
paper_authors._id: 64927a832aa84c46de6ba8f4
paper_authors.name: Zsolt Kira
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:36:16.491Z
paper_authors._id: 64716e6e085607e2f5842446
paper_authors.avatarUrl: /avatars/f69a0869eb6adfdbfd746a56390de2df.svg
paper_authors.fullname: Zsolt Kira
paper_authors.type: user
paper_authors.user: zk-danti
paper_authors._id: 64927a832aa84c46de6ba8f5
paper_authors.name: Manolis Savva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:37:04.370Z
paper_authors._id: 639f45949f1f2baab2f34c79
paper_authors.avatarUrl: /avatars/6035ba3b8ccecda262316076dfd1ff60.svg
paper_authors.fullname: Manolis Savva
paper_authors.type: user
paper_authors.user: msavva
paper_authors._id: 64927a832aa84c46de6ba8f6
paper_authors.name: Angel Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:37:25.335Z
paper_authors._id: 6363d5d92e11f41bc819176e
paper_authors.avatarUrl: /avatars/516bea7f7da973bfcc992df9a63727ca.svg
paper_authors.fullname: Angel Chang
paper_authors.type: user
paper_authors.user: angelxuanchang
paper_authors._id: 64927a832aa84c46de6ba8f7
paper_authors.name: Devendra Singh Chaplot
paper_authors._id: 64927a832aa84c46de6ba8f8
paper_authors.name: Dhruv Batra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:38:02.559Z
paper_authors._id: 63ed1e57f765928ceeacbe02
paper_authors.avatarUrl: /avatars/e5a7428bb0acb43a240f1c4a8c1a5639.svg
paper_authors.fullname: Dhruv Batra
paper_authors.type: user
paper_authors.user: dbatra
paper_authors._id: 64927a832aa84c46de6ba8f9
paper_authors.name: Roozbeh Mottaghi
paper_authors._id: 64927a832aa84c46de6ba8fa
paper_authors.name: Yonatan Bisk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T15:38:36.509Z
paper_authors._id: 63180f3c879073cf83f135b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63180f3c879073cf83f135b0/0ycXVKOywlU1hcvQ4MDws.jpeg
paper_authors.fullname: Yonatan Bisk
paper_authors.type: user
paper_authors.user: ybisk
paper_authors._id: 64927a832aa84c46de6ba8fb
paper_authors.name: Chris Paxton
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-21T20:46:30.825Z
paper_authors._id: 62b13b28b7053ab38d2d5f96
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b13b28b7053ab38d2d5f96/V6780Omima7sNW5E4TEeW.png
paper_authors.fullname: Chris Paxton
paper_authors.type: user
paper_authors.user: cpaxton
paper_title: HomeRobot: Open-Vocabulary Mobile Manipulation
paper_summary: HomeRobot (noun): An affordable compliant robot that navigates homes and
  manipulates a wide range of objects in order to complete everyday tasks.
  Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object
  in any unseen environment, and placing it in a commanded location. This is a
  foundational challenge for robots to be useful assistants in human
  environments, because it involves tackling sub-problems from across robotics:
  perception, language understanding, navigation, and manipulation are all
  essential to OVMM. In addition, integration of the solutions to these
  sub-problems poses its own substantial challenges. To drive research in this
  area, we introduce the HomeRobot OVMM benchmark, where an agent navigates
  household environments to grasp novel objects and place them on target
  receptacles. HomeRobot has two components: a simulation component, which uses a
  large and diverse curated object set in new, high-quality multi-room home
  environments; and a real-world component, providing a software stack for the
  low-cost Hello Robot Stretch to encourage replication of real-world experiments
  across labs. We implement both reinforcement learning and heuristic
  (model-based) baselines and show evidence of sim-to-real transfer. Our
  baselines achieve a 20% success rate in the real world; our experiments
  identify ways future research work improve performance. See videos on our
  website: https://ovmm.github.io/.
paper_ai_keywords: reinforcement learning heuristic model-based sim-to-real transfer