title: WildLMa: Long Horizon Loco-Manipulation in the Wild
summary: `In-the-wild' mobile manipulation aims to deploy robots in diverse real-world
  environments, which requires the robot to (1) have skills that generalize
  across object configurations; (2) be capable of long-horizon task execution in
  diverse environments; and (3) perform complex manipulation beyond
  pick-and-place. Quadruped robots with manipulators hold promise for extending
  the workspace and enabling robust locomotion, but existing results do not
  investigate such a capability. This paper proposes WildLMa with three
  components to address these issues: (1) adaptation of learned low-level
  controller for VR-enabled whole-body teleoperation and traversability; (2)
  WildLMa-Skill -- a library of generalizable visuomotor skills acquired via
  imitation learning or heuristics and (3) WildLMa-Planner -- an interface of
  learned skills that allow LLM planners to coordinate skills for long-horizon
  tasks. We demonstrate the importance of high-quality training data by achieving
  higher grasping success rate over existing RL baselines using only tens of
  demonstrations. WildLMa exploits CLIP for language-conditioned imitation
  learning that empirically generalizes to objects unseen in training
  demonstrations. Besides extensive quantitative evaluation, we qualitatively
  demonstrate practical robot applications, such as cleaning up trash in
  university hallways or outdoor terrains, operating articulated objects, and
  rearranging items on a bookshelf.
paper_authors._id: 674413c5b5259d64b54af07b
paper_authors.name: Ri-Zhao Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T21:38:57.101Z
paper_authors._id: 64eced8b6343a356327f5b45
paper_authors.avatarUrl: /avatars/cf9de701a6fa893c7989773c4875c254.svg
paper_authors.fullname: Roger Qiu
paper_authors.type: user
paper_authors.user: RogerQi
paper_authors._id: 674413c5b5259d64b54af07c
paper_authors.name: Yuchen Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T21:18:12.708Z
paper_authors._id: 65482cb95f6c84d8f4c2267e
paper_authors.avatarUrl: /avatars/c047bbb89db5bafe12f51e88a6f6dd0c.svg
paper_authors.fullname: Yuchen Song
paper_authors.type: user
paper_authors.user: YuchenEasonSong
paper_authors._id: 674413c5b5259d64b54af07d
paper_authors.name: Xuanbin Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T20:39:15.857Z
paper_authors._id: 66f717cce5d2d88a93fe33a5
paper_authors.avatarUrl: /avatars/dc9824ba08db7be050df6ba97e1da84c.svg
paper_authors.fullname: Xuanbin Peng
paper_authors.type: user
paper_authors.user: xuanbin-peng
paper_authors._id: 674413c5b5259d64b54af07e
paper_authors.name: Sai Aneesh Suryadevara
paper_authors._id: 674413c5b5259d64b54af07f
paper_authors.name: Ge Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:07:41.294Z
paper_authors._id: 5f03ee98ff69163f6f59e9e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594093169905-noauth.jpeg
paper_authors.fullname: Ge Yang
paper_authors.type: user
paper_authors.user: geyang
paper_authors._id: 674413c5b5259d64b54af080
paper_authors.name: Minghuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:01:08.405Z
paper_authors._id: 630c8f0c8e3ff0c72327140a
paper_authors.avatarUrl: /avatars/998de36d102b8bec85624f209272f7f2.svg
paper_authors.fullname: Minghuan Liu
paper_authors.type: user
paper_authors.user: ericonaldo
paper_authors._id: 674413c5b5259d64b54af081
paper_authors.name: Mazeyu Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:07:19.940Z
paper_authors._id: 657eba2cf4f72f2c4c3d3862
paper_authors.avatarUrl: /avatars/51624276469091758c2347147bb5cc4b.svg
paper_authors.fullname: Mazeyu Ji
paper_authors.type: user
paper_authors.user: jimazeyu
paper_authors._id: 674413c5b5259d64b54af082
paper_authors.name: Chengzhe Jia
paper_authors._id: 674413c5b5259d64b54af083
paper_authors.name: Ruihan Yang
paper_authors._id: 674413c5b5259d64b54af084
paper_authors.name: Xueyan Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:06:43.964Z
paper_authors._id: 62520988818a5dc29ab91d6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
paper_authors.fullname: Xueyan Zou
paper_authors.type: user
paper_authors.user: xueyanz
paper_authors._id: 674413c5b5259d64b54af085
paper_authors.name: Xiaolong Wang
paper_title: WildLMa: Long Horizon Loco-Manipulation in the Wild
paper_summary: `In-the-wild' mobile manipulation aims to deploy robots in diverse real-world
  environments, which requires the robot to (1) have skills that generalize
  across object configurations; (2) be capable of long-horizon task execution in
  diverse environments; and (3) perform complex manipulation beyond
  pick-and-place. Quadruped robots with manipulators hold promise for extending
  the workspace and enabling robust locomotion, but existing results do not
  investigate such a capability. This paper proposes WildLMa with three
  components to address these issues: (1) adaptation of learned low-level
  controller for VR-enabled whole-body teleoperation and traversability; (2)
  WildLMa-Skill -- a library of generalizable visuomotor skills acquired via
  imitation learning or heuristics and (3) WildLMa-Planner -- an interface of
  learned skills that allow LLM planners to coordinate skills for long-horizon
  tasks. We demonstrate the importance of high-quality training data by achieving
  higher grasping success rate over existing RL baselines using only tens of
  demonstrations. WildLMa exploits CLIP for language-conditioned imitation
  learning that empirically generalizes to objects unseen in training
  demonstrations. Besides extensive quantitative evaluation, we qualitatively
  demonstrate practical robot applications, such as cleaning up trash in
  university hallways or outdoor terrains, operating articulated objects, and
  rearranging items on a bookshelf.
paper_ai_summary: WildLMa integrates learned low-level controllers, a skill library, and a planner to enable quadruped robots to perform complex manipulation tasks in diverse environments with minimal training data.
paper_ai_keywords: low-level controller VR-enabled whole-body teleoperation traversability visuomotor skills imitation learning heuristics learned skills LLM planners long-horizon tasks language-conditioned imitation learning CLIP grasping success rate RL baselines