title: Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual
    and Action Representations
summary: Autonomous robotic systems capable of learning novel manipulation tasks are
  poised to transform industries from manufacturing to service automation.
  However, modern methods (e.g., VIP and R3M) still face significant hurdles,
  notably the domain gap among robotic embodiments and the sparsity of successful
  task executions within specific action spaces, resulting in misaligned and
  ambiguous task representations. We introduce Ag2Manip (Agent-Agnostic
  representations for Manipulation), a framework aimed at surmounting these
  challenges through two key innovations: a novel agent-agnostic visual
  representation derived from human manipulation videos, with the specifics of
  embodiments obscured to enhance generalizability; and an agent-agnostic action
  representation abstracting a robot's kinematics to a universal agent proxy,
  emphasizing crucial interactions between end-effector and object. Ag2Manip's
  empirical validation across simulated benchmarks like FrankaKitchen, ManiSkill,
  and PartManip shows a 325% increase in performance, achieved without
  domain-specific demonstrations. Ablation studies underline the essential
  contributions of the visual and action representations to this success.
  Extending our evaluations to the real world, Ag2Manip significantly improves
  imitation learning success rates from 50% to 77.5%, demonstrating its
  effectiveness and generalizability across both simulated and physical
  environments.
paper_authors._id: 66305ba8090586ab8c761440
paper_authors.name: Puhao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:55:03.163Z
paper_authors._id: 63d4e83913ae45b7807acbb1
paper_authors.avatarUrl: /avatars/4b8778ed239b64de5f8781f0ba57168b.svg
paper_authors.fullname: Puhao Li
paper_authors.type: user
paper_authors.user: puhao
paper_authors._id: 66305ba8090586ab8c761441
paper_authors.name: Tengyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:55:23.988Z
paper_authors._id: 634379a81bdd3dfa55dcbe82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665366425318-noauth.png
paper_authors.fullname: Tengyu Liu
paper_authors.type: user
paper_authors.user: oxrider
paper_authors._id: 66305ba8090586ab8c761442
paper_authors.name: Yuyang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:55:48.311Z
paper_authors._id: 637cef70b8e573d75beb5748
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637cef70b8e573d75beb5748/hMl_pKUO3BanR47RQqN6o.png
paper_authors.fullname: Yuyang Li
paper_authors.type: user
paper_authors.user: aidenli
paper_authors._id: 66305ba8090586ab8c761443
paper_authors.name: Muzhi Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:55:55.002Z
paper_authors._id: 64ba3db4b4f1dd6959da9367
paper_authors.avatarUrl: /avatars/41d13aef18ef79abab0c6a8e6037b458.svg
paper_authors.fullname: Muzhi Han
paper_authors.type: user
paper_authors.user: hamster6
paper_authors._id: 66305ba8090586ab8c761444
paper_authors.name: Haoran Geng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:56:00.738Z
paper_authors._id: 651a752df7724d9aee9fb970
paper_authors.avatarUrl: /avatars/956dc2a6fabf10bea15994152c04badc.svg
paper_authors.fullname: Haoran Geng
paper_authors.type: user
paper_authors.user: HaoranGeng
paper_authors._id: 66305ba8090586ab8c761445
paper_authors.name: Shu Wang
paper_authors._id: 66305ba8090586ab8c761446
paper_authors.name: Yixin Zhu
paper_authors._id: 66305ba8090586ab8c761447
paper_authors.name: Song-Chun Zhu
paper_authors._id: 66305ba8090586ab8c761448
paper_authors.name: Siyuan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:05:42.906Z
paper_authors._id: 63c7a33121bd95f80ed74652
paper_authors.avatarUrl: /avatars/7dd59afea785a2bff0ec2b757abd474e.svg
paper_authors.fullname: Siyuan Huang
paper_authors.type: user
paper_authors.user: thuhsy
paper_title: Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual
    and Action Representations
paper_summary: Autonomous robotic systems capable of learning novel manipulation tasks are
  poised to transform industries from manufacturing to service automation.
  However, modern methods (e.g., VIP and R3M) still face significant hurdles,
  notably the domain gap among robotic embodiments and the sparsity of successful
  task executions within specific action spaces, resulting in misaligned and
  ambiguous task representations. We introduce Ag2Manip (Agent-Agnostic
  representations for Manipulation), a framework aimed at surmounting these
  challenges through two key innovations: a novel agent-agnostic visual
  representation derived from human manipulation videos, with the specifics of
  embodiments obscured to enhance generalizability; and an agent-agnostic action
  representation abstracting a robot's kinematics to a universal agent proxy,
  emphasizing crucial interactions between end-effector and object. Ag2Manip's
  empirical validation across simulated benchmarks like FrankaKitchen, ManiSkill,
  and PartManip shows a 325% increase in performance, achieved without
  domain-specific demonstrations. Ablation studies underline the essential
  contributions of the visual and action representations to this success.
  Extending our evaluations to the real world, Ag2Manip significantly improves
  imitation learning success rates from 50% to 77.5%, demonstrating its
  effectiveness and generalizability across both simulated and physical
  environments.
paper_ai_keywords: VIP R3M domain gap action spaces task representations Ag2Manip (Agent-Agnostic representations for Manipulation) visual representation human manipulation videos agent-agnostic end-effector object FrankaKitchen ManiSkill PartManip imitation learning