Learning Generalizable Feature Fields for Mobile Manipulation
<|reference_start|>Learning Generalizable Feature Fields for Mobile Manipulation: An open problem in mobile manipulation is how to represent objects and scenes in a unified manner, so that robots can use it both for navigating in the environment and manipulating objects. The latter requires capturing intricate geometry while understanding fine-grained semantics, whereas the former involves capturing the complexity inherit to an expansive physical scale. In this work, we present GeFF (Generalizable Feature Fields), a scene-level generalizable neural feature field that acts as a unified representation for both navigation and manipulation that performs in real-time. To do so, we treat generative novel view synthesis as a pre-training task, and then align the resulting rich scene priors with natural language via CLIP feature distillation. We demonstrate the effectiveness of this approach by deploying GeFF on a quadrupedal robot equipped with a manipulator. We evaluate GeFF's ability to generalize to open-set objects as well as running time, when performing open-vocabulary mobile manipulation in dynamic scenes.<|reference_end|>
@article{qiu2024learning,
    title={Learning Generalizable Feature Fields for Mobile Manipulation},
    author={Ri-Zhao Qiu, Yafei Hu, Ge Yang, Yuchen Song, Yang Fu, Jianglong Ye,
  Jiteng Mu, Ruihan Yang, Nikolay Atanasov, Sebastian Scherer, Xiaolong Wang},
    journal={arXiv preprint arXiv:2403.07563},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.07563},
    primaryClass={cs.RO cs.CV cs.LG}
}