title: Helpful DoggyBot: Open-World Object Fetching using Legged Robots and
    Vision-Language Models
summary: Learning-based methods have achieved strong performance for quadrupedal
  locomotion. However, several challenges prevent quadrupeds from learning
  helpful indoor skills that require interaction with environments and humans:
  lack of end-effectors for manipulation, limited semantic understanding using
  only simulation data, and low traversability and reachability in indoor
  environments. We present a system for quadrupedal mobile manipulation in indoor
  environments. It uses a front-mounted gripper for object manipulation, a
  low-level controller trained in simulation using egocentric depth for agile
  skills like climbing and whole-body tilting, and pre-trained vision-language
  models (VLMs) with a third-person fisheye and an egocentric RGB camera for
  semantic understanding and command generation. We evaluate our system in two
  unseen environments without any real-world data collection or training. Our
  system can zero-shot generalize to these environments and complete tasks, like
  following user's commands to fetch a randomly placed stuff toy after climbing
  over a queen-sized bed, with a 60% success rate. Project website:
  https://helpful-doggybot.github.io/
paper_authors._id: 66fcb48725ceeb69c30537ea
paper_authors.name: Qi Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:44:34.199Z
paper_authors._id: 670f6cbda4e6ac0ef673530a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mtrGsHBXXC1hS0BmWDGio.png
paper_authors.fullname: Qi Wu
paper_authors.type: user
paper_authors.user: wooqi
paper_authors._id: 66fcb48725ceeb69c30537eb
paper_authors.name: Zipeng Fu
paper_authors._id: 66fcb48725ceeb69c30537ec
paper_authors.name: Xuxin Cheng
paper_authors._id: 66fcb48725ceeb69c30537ed
paper_authors.name: Xiaolong Wang
paper_authors._id: 66fcb48725ceeb69c30537ee
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T09:04:20.818Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_title: Helpful DoggyBot: Open-World Object Fetching using Legged Robots and
    Vision-Language Models
paper_summary: Learning-based methods have achieved strong performance for quadrupedal
  locomotion. However, several challenges prevent quadrupeds from learning
  helpful indoor skills that require interaction with environments and humans:
  lack of end-effectors for manipulation, limited semantic understanding using
  only simulation data, and low traversability and reachability in indoor
  environments. We present a system for quadrupedal mobile manipulation in indoor
  environments. It uses a front-mounted gripper for object manipulation, a
  low-level controller trained in simulation using egocentric depth for agile
  skills like climbing and whole-body tilting, and pre-trained vision-language
  models (VLMs) with a third-person fisheye and an egocentric RGB camera for
  semantic understanding and command generation. We evaluate our system in two
  unseen environments without any real-world data collection or training. Our
  system can zero-shot generalize to these environments and complete tasks, like
  following user's commands to fetch a randomly placed stuff toy after climbing
  over a queen-sized bed, with a 60% success rate. Project website:
  https://helpful-doggybot.github.io/
paper_ai_summary: A quadruped mobile manipulation system uses a gripper, simulation-trained low-level controller, and vision-language models for object manipulation and command understanding in indoor environments.
paper_ai_keywords: front-mounted gripper low-level controller egocentric depth pre-trained vision-language models third-person fisheye camera egocentric RGB camera semantic understanding zero-shot generalization