title: Learning Generalizable Feature Fields for Mobile Manipulation
summary: An open problem in mobile manipulation is how to represent objects and scenes
  in a unified manner, so that robots can use it both for navigating in the
  environment and manipulating objects. The latter requires capturing intricate
  geometry while understanding fine-grained semantics, whereas the former
  involves capturing the complexity inherit to an expansive physical scale. In
  this work, we present GeFF (Generalizable Feature Fields), a scene-level
  generalizable neural feature field that acts as a unified representation for
  both navigation and manipulation that performs in real-time. To do so, we treat
  generative novel view synthesis as a pre-training task, and then align the
  resulting rich scene priors with natural language via CLIP feature
  distillation. We demonstrate the effectiveness of this approach by deploying
  GeFF on a quadrupedal robot equipped with a manipulator. We evaluate GeFF's
  ability to generalize to open-set objects as well as running time, when
  performing open-vocabulary mobile manipulation in dynamic scenes.
paper_authors._id: 65f11bae6750342ce0915664
paper_authors.name: Ri-Zhao Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T21:38:54.375Z
paper_authors._id: 64eced8b6343a356327f5b45
paper_authors.avatarUrl: /avatars/cf9de701a6fa893c7989773c4875c254.svg
paper_authors.fullname: Roger Qiu
paper_authors.type: user
paper_authors.user: RogerQi
paper_authors._id: 65f11bae6750342ce0915665
paper_authors.name: Yafei Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-13T13:26:27.653Z
paper_authors._id: 63f3f1dca096536aeab25fcc
paper_authors.avatarUrl: /avatars/7de71ca0c1b6711f4245fa75a0886225.svg
paper_authors.fullname: Yafei Hu
paper_authors.type: user
paper_authors.user: jeffreyhu
paper_authors._id: 65f11bae6750342ce0915666
paper_authors.name: Ge Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-13T13:59:01.984Z
paper_authors._id: 5f03ee98ff69163f6f59e9e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594093169905-noauth.jpeg
paper_authors.fullname: Ge Yang
paper_authors.type: user
paper_authors.user: geyang
paper_authors._id: 65f11bae6750342ce0915667
paper_authors.name: Yuchen Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T21:18:11.077Z
paper_authors._id: 65482cb95f6c84d8f4c2267e
paper_authors.avatarUrl: /avatars/c047bbb89db5bafe12f51e88a6f6dd0c.svg
paper_authors.fullname: Yuchen Song
paper_authors.type: user
paper_authors.user: YuchenEasonSong
paper_authors._id: 65f11bae6750342ce0915668
paper_authors.name: Yang Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-13T13:59:46.939Z
paper_authors._id: 638d9a02a11654155ba9c7cb
paper_authors.avatarUrl: /avatars/db85a28500554c1989b5f744bcd8b220.svg
paper_authors.fullname: Yang Fu
paper_authors.type: user
paper_authors.user: yangfu2
paper_authors._id: 65f11bae6750342ce0915669
paper_authors.name: Jianglong Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-13T13:59:53.438Z
paper_authors._id: 634e2696d00bb5d92c3f4e86
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674216878324-634e2696d00bb5d92c3f4e86.jpeg
paper_authors.fullname: Jianglong Ye
paper_authors.type: user
paper_authors.user: jianglongye
paper_authors._id: 65f11bae6750342ce091566a
paper_authors.name: Jiteng Mu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-13T14:00:10.024Z
paper_authors._id: 63a38a4a2262903957e12f14
paper_authors.avatarUrl: /avatars/ad0e98736bdba5fa7a64c15c876c67a4.svg
paper_authors.fullname: JitengMu
paper_authors.type: user
paper_authors.user: JitengMu
paper_authors._id: 65f11bae6750342ce091566b
paper_authors.name: Ruihan Yang
paper_authors._id: 65f11bae6750342ce091566c
paper_authors.name: Nikolay Atanasov
paper_authors._id: 65f11bae6750342ce091566d
paper_authors.name: Sebastian Scherer
paper_authors._id: 65f11bae6750342ce091566e
paper_authors.name: Xiaolong Wang
paper_title: Learning Generalizable Feature Fields for Mobile Manipulation
paper_summary: An open problem in mobile manipulation is how to represent objects and scenes
  in a unified manner, so that robots can use it both for navigating in the
  environment and manipulating objects. The latter requires capturing intricate
  geometry while understanding fine-grained semantics, whereas the former
  involves capturing the complexity inherit to an expansive physical scale. In
  this work, we present GeFF (Generalizable Feature Fields), a scene-level
  generalizable neural feature field that acts as a unified representation for
  both navigation and manipulation that performs in real-time. To do so, we treat
  generative novel view synthesis as a pre-training task, and then align the
  resulting rich scene priors with natural language via CLIP feature
  distillation. We demonstrate the effectiveness of this approach by deploying
  GeFF on a quadrupedal robot equipped with a manipulator. We evaluate GeFF's
  ability to generalize to open-set objects as well as running time, when
  performing open-vocabulary mobile manipulation in dynamic scenes.
paper_ai_summary: GeFF, a generalizable neural feature field, unifies navigation and manipulation tasks by leveraging generative novel view synthesis and CLIP feature distillation for real-time performance in dynamic scenes.
paper_ai_keywords: Generalizable Feature Fields neural feature field generative novel view synthesis CLIP feature distillation quadrupedal robot open-set objects open-vocabulary mobile manipulation