title: Learning to Manipulate Anywhere: A Visual Generalizable Framework For
    Reinforcement Learning
summary: Can we endow visuomotor robots with generalization capabilities to operate in
  diverse open-world scenarios? In this paper, we propose Maniwhere, a
  generalizable framework tailored for visual reinforcement learning, enabling
  the trained robot policies to generalize across a combination of multiple
  visual disturbance types. Specifically, we introduce a multi-view
  representation learning approach fused with Spatial Transformer Network (STN)
  module to capture shared semantic information and correspondences among
  different viewpoints. In addition, we employ a curriculum-based randomization
  and augmentation approach to stabilize the RL training process and strengthen
  the visual generalization ability. To exhibit the effectiveness of Maniwhere,
  we meticulously design 8 tasks encompassing articulate objects, bi-manual, and
  dexterous hand manipulation tasks, demonstrating Maniwhere's strong visual
  generalization and sim2real transfer abilities across 3 hardware platforms. Our
  experiments show that Maniwhere significantly outperforms existing
  state-of-the-art methods. Videos are provided at
  https://gemcollector.github.io/maniwhere/.
paper_authors._id: 66a1c2d437b81357eff321f9
paper_authors.name: Zhecheng Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-25T07:38:24.557Z
paper_authors._id: 64a0dd823886663ff8455f7a
paper_authors.avatarUrl: /avatars/a8560f9c9de59cbab4ef5ed9494e4051.svg
paper_authors.fullname: Zhecheng Yuan
paper_authors.type: user
paper_authors.user: gemcollector
paper_authors._id: 66a1c2d437b81357eff321fa
paper_authors.name: Tianming Wei
paper_authors._id: 66a1c2d437b81357eff321fb
paper_authors.name: Shuiqi Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T12:25:15.728Z
paper_authors._id: 65f493d1a8d2ff8f8f94689d
paper_authors.avatarUrl: /avatars/4f01775f79be964d4ee4a6d82b3b5943.svg
paper_authors.fullname: shuiqi cheng
paper_authors.type: user
paper_authors.user: shuiqicheng
paper_authors._id: 66a1c2d437b81357eff321fc
paper_authors.name: Gu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T12:25:33.952Z
paper_authors._id: 65e95effe19214e9d13d7a2a
paper_authors.avatarUrl: /avatars/35ae40579ac53d7b598957be9c8b4710.svg
paper_authors.fullname: Gu Zhang
paper_authors.type: user
paper_authors.user: GuZhang
paper_authors._id: 66a1c2d437b81357eff321fd
paper_authors.name: Yuanpei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T12:25:42.807Z
paper_authors._id: 6393a8af84c565d2c3419b7c
paper_authors.avatarUrl: /avatars/f2a237a58dd0a25ef5c1a98e60acbb5c.svg
paper_authors.fullname: chen
paper_authors.type: user
paper_authors.user: yuanpei
paper_authors._id: 66a1c2d437b81357eff321fe
paper_authors.name: Huazhe Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T12:25:48.615Z
paper_authors._id: 649c130f92525b3ac9fbcffe
paper_authors.avatarUrl: /avatars/b60e4d6803f7e4ef42200ddcb9e5bac1.svg
paper_authors.fullname: Huazhe Xu
paper_authors.type: user
paper_authors.user: Huazhe-Xu
paper_title: Learning to Manipulate Anywhere: A Visual Generalizable Framework For
    Reinforcement Learning
paper_summary: Can we endow visuomotor robots with generalization capabilities to operate in
  diverse open-world scenarios? In this paper, we propose Maniwhere, a
  generalizable framework tailored for visual reinforcement learning, enabling
  the trained robot policies to generalize across a combination of multiple
  visual disturbance types. Specifically, we introduce a multi-view
  representation learning approach fused with Spatial Transformer Network (STN)
  module to capture shared semantic information and correspondences among
  different viewpoints. In addition, we employ a curriculum-based randomization
  and augmentation approach to stabilize the RL training process and strengthen
  the visual generalization ability. To exhibit the effectiveness of Maniwhere,
  we meticulously design 8 tasks encompassing articulate objects, bi-manual, and
  dexterous hand manipulation tasks, demonstrating Maniwhere's strong visual
  generalization and sim2real transfer abilities across 3 hardware platforms. Our
  experiments show that Maniwhere significantly outperforms existing
  state-of-the-art methods. Videos are provided at
  https://gemcollector.github.io/maniwhere/.
paper_ai_keywords: visual reinforcement learning multi-view representation learning Spatial Transformer Network (STN) curriculum-based randomization augmentation approach articulate objects bi-manual manipulation dexterous hand manipulation sim2real transfer