Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning
<|reference_start|>Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning: Can we endow visuomotor robots with generalization capabilities to operate in diverse open-world scenarios? In this paper, we propose \textbf{Maniwhere}, a generalizable framework tailored for visual reinforcement learning, enabling the trained robot policies to generalize across a combination of multiple visual disturbance types. Specifically, we introduce a multi-view representation learning approach fused with Spatial Transformer Network (STN) module to capture shared semantic information and correspondences among different viewpoints. In addition, we employ a curriculum-based randomization and augmentation approach to stabilize the RL training process and strengthen the visual generalization ability. To exhibit the effectiveness of Maniwhere, we meticulously design 8 tasks encompassing articulate objects, bi-manual, and dexterous hand manipulation tasks, demonstrating Maniwhere's strong visual generalization and sim2real transfer abilities across 3 hardware platforms. Our experiments show that Maniwhere significantly outperforms existing state-of-the-art methods. Videos are provided at https://gemcollector.github.io/maniwhere/.<|reference_end|>
@article{yuan2024learning,
    title={Learning to Manipulate Anywhere: A Visual Generalizable Framework For
  Reinforcement Learning},
    author={Zhecheng Yuan, Tianming Wei, Shuiqi Cheng, Gu Zhang, Yuanpei Chen,
  Huazhe Xu},
    journal={arXiv preprint arXiv:2407.15815},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.15815},
    primaryClass={cs.RO cs.AI cs.CV}
}