title: OK-Robot: What Really Matters in Integrating Open-Knowledge Models for
    Robotics
summary: Remarkable progress has been made in recent years in the fields of vision,
  language, and robotics. We now have vision models capable of recognizing
  objects based on language queries, navigation systems that can effectively
  control mobile systems, and grasping models that can handle a wide range of
  objects. Despite these advancements, general-purpose applications of robotics
  still lag behind, even though they rely on these fundamental capabilities of
  recognition, navigation, and grasping. In this paper, we adopt a systems-first
  approach to develop a new Open Knowledge-based robotics framework called
  OK-Robot. By combining Vision-Language Models (VLMs) for object detection,
  navigation primitives for movement, and grasping primitives for object
  manipulation, OK-Robot offers a integrated solution for pick-and-drop
  operations without requiring any training. To evaluate its performance, we run
  OK-Robot in 10 real-world home environments. The results demonstrate that
  OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks,
  representing a new state-of-the-art in Open Vocabulary Mobile Manipulation
  (OVMM) with nearly 1.8x the performance of prior work. On cleaner, uncluttered
  environments, OK-Robot's performance increases to 82%. However, the most
  important insight gained from OK-Robot is the critical role of nuanced details
  when combining Open Knowledge systems like VLMs with robotic modules. Videos of
  our experiments are available on our website: https://ok-robot.github.io
paper_authors._id: 65af3fa852e1b2aae41667a7
paper_authors.name: Peiqi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T10:53:51.316Z
paper_authors._id: 631f1793c1a8269da393eb15
paper_authors.avatarUrl: /avatars/3cb07f9ef3f7f1298da7a880661b6689.svg
paper_authors.fullname: Peiqi Liu
paper_authors.type: user
paper_authors.user: LiuPeiqi
paper_authors._id: 65af3fa852e1b2aae41667a8
paper_authors.name: Yaswanth Orru
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T10:53:58.227Z
paper_authors._id: 655c6db7419819fec2cf85b2
paper_authors.avatarUrl: /avatars/7de3a409d15fd60b44032bceadf2c74a.svg
paper_authors.fullname: Yaswanth Orru
paper_authors.type: user
paper_authors.user: yaswantho
paper_authors._id: 65af3fa852e1b2aae41667a9
paper_authors.name: Chris Paxton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T10:54:04.886Z
paper_authors._id: 62b13b28b7053ab38d2d5f96
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b13b28b7053ab38d2d5f96/V6780Omima7sNW5E4TEeW.png
paper_authors.fullname: Chris Paxton
paper_authors.type: user
paper_authors.user: cpaxton
paper_authors._id: 65af3fa852e1b2aae41667aa
paper_authors.name: Nur Muhammad Mahi Shafiullah
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T08:08:54.262Z
paper_authors._id: 630e567f8df86f1e5bf0d837
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630e567f8df86f1e5bf0d837/Mje-2ahgmMET2jzqLafhy.jpeg
paper_authors.fullname: Mahi Shafiullah
paper_authors.type: user
paper_authors.user: notmahi
paper_authors._id: 65af3fa852e1b2aae41667ab
paper_authors.name: Lerrel Pinto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T10:54:11.294Z
paper_authors._id: 6388cdb6685efaedd15dc0f3
paper_authors.avatarUrl: /avatars/b2b74454bf86bf61fd90ba088644f547.svg
paper_authors.fullname: Lerrel Pinto
paper_authors.type: user
paper_authors.user: lerrel
paper_title: OK-Robot: What Really Matters in Integrating Open-Knowledge Models for
    Robotics
paper_summary: Remarkable progress has been made in recent years in the fields of vision,
  language, and robotics. We now have vision models capable of recognizing
  objects based on language queries, navigation systems that can effectively
  control mobile systems, and grasping models that can handle a wide range of
  objects. Despite these advancements, general-purpose applications of robotics
  still lag behind, even though they rely on these fundamental capabilities of
  recognition, navigation, and grasping. In this paper, we adopt a systems-first
  approach to develop a new Open Knowledge-based robotics framework called
  OK-Robot. By combining Vision-Language Models (VLMs) for object detection,
  navigation primitives for movement, and grasping primitives for object
  manipulation, OK-Robot offers a integrated solution for pick-and-drop
  operations without requiring any training. To evaluate its performance, we run
  OK-Robot in 10 real-world home environments. The results demonstrate that
  OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks,
  representing a new state-of-the-art in Open Vocabulary Mobile Manipulation
  (OVMM) with nearly 1.8x the performance of prior work. On cleaner, uncluttered
  environments, OK-Robot's performance increases to 82%. However, the most
  important insight gained from OK-Robot is the critical role of nuanced details
  when combining Open Knowledge systems like VLMs with robotic modules. Videos of
  our experiments are available on our website: https://ok-robot.github.io
paper_ai_summary: OK-Robot, a new framework combining Vision-Language Models, navigation primitives, and grasping primitives, achieves high success rates in open-ended pick-and-drop tasks across various home environments.
paper_ai_keywords: Vision-Language Models object detection navigation primitives grasping primitives Open Vocabulary Mobile Manipulation open-ended pick-and-drop tasks