RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation
<|reference_start|>RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation: This work proposes a retrieve-and-transfer framework for zero-shot robotic manipulation, dubbed RAM, featuring generalizability across various objects, environments, and embodiments. Unlike existing approaches that learn manipulation from expensive in-domain demonstrations, RAM capitalizes on a retrieval-based affordance transfer paradigm to acquire versatile manipulation capabilities from abundant out-of-domain data. First, RAM extracts unified affordance at scale from diverse sources of demonstrations including robotic data, human-object interaction (HOI) data, and custom data to construct a comprehensive affordance memory. Then given a language instruction, RAM hierarchically retrieves the most similar demonstration from the affordance memory and transfers such out-of-domain 2D affordance to in-domain 3D executable affordance in a zero-shot and embodiment-agnostic manner. Extensive simulation and real-world evaluations demonstrate that our RAM consistently outperforms existing works in diverse daily tasks. Additionally, RAM shows significant potential for downstream applications such as automatic and efficient data collection, one-shot visual imitation, and LLM/VLM-integrated long-horizon manipulation. For more details, please check our website at https://yxkryptonite.github.io/RAM/.<|reference_end|>
@article{kuang2024ram:,
    title={RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot
  Robotic Manipulation},
    author={Yuxuan Kuang, Junjie Ye, Haoran Geng, Jiageng Mao, Congyue Deng,
  Leonidas Guibas, He Wang, Yue Wang},
    journal={arXiv preprint arXiv:2407.04689},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.04689},
    primaryClass={cs.RO cs.CV}
}