title: Robots Pre-train Robots: Manipulation-Centric Robotic Representation
    from Large-Scale Robot Dataset
summary: The pre-training of visual representations has enhanced the efficiency of
  robot learning. Due to the lack of large-scale in-domain robotic datasets,
  prior works utilize in-the-wild human videos to pre-train robotic visual
  representation. Despite their promising results, representations from human
  videos are inevitably subject to distribution shifts and lack the dynamics
  information crucial for task completion. We first evaluate various pre-trained
  representations in terms of their correlation to the downstream robotic
  manipulation tasks (i.e., manipulation centricity). Interestingly, we find that
  the "manipulation centricity" is a strong indicator of success rates when
  applied to downstream tasks. Drawing from these findings, we propose
  Manipulation Centric Representation (MCR), a foundation representation learning
  framework capturing both visual features and the dynamics information such as
  actions and proprioceptions of manipulation tasks to improve manipulation
  centricity. Specifically, we pre-train a visual encoder on the DROID robotic
  dataset and leverage motion-relevant data such as robot proprioceptive states
  and actions. We introduce a novel contrastive loss that aligns visual
  observations with the robot's proprioceptive state-action dynamics, combined
  with a behavior cloning (BC)-like actor loss to predict actions during
  pre-training, along with a time contrastive loss. Empirical results across 4
  simulation domains with 20 tasks verify that MCR outperforms the strongest
  baseline method by 14.8%. Moreover, MCR boosts the performance of
  data-efficient learning with a UR5e arm on 3 real-world tasks by 76.9%. Project
  website: https://robots-pretrain-robots.github.io/.
paper_authors._id: 67219f7ee14a54a801766802
paper_authors.name: Guangqi Jiang
paper_authors._id: 67219f7ee14a54a801766803
paper_authors.name: Yifei Sun
paper_authors._id: 67219f7ee14a54a801766804
paper_authors.name: Tao Huang
paper_authors._id: 67219f7ee14a54a801766805
paper_authors.name: Huanyu Li
paper_authors._id: 67219f7ee14a54a801766806
paper_authors.name: Yongyuan Liang
paper_authors._id: 67219f7ee14a54a801766807
paper_authors.name: Huazhe Xu
paper_title: Robots Pre-train Robots: Manipulation-Centric Robotic Representation
    from Large-Scale Robot Dataset
paper_summary: The pre-training of visual representations has enhanced the efficiency of
  robot learning. Due to the lack of large-scale in-domain robotic datasets,
  prior works utilize in-the-wild human videos to pre-train robotic visual
  representation. Despite their promising results, representations from human
  videos are inevitably subject to distribution shifts and lack the dynamics
  information crucial for task completion. We first evaluate various pre-trained
  representations in terms of their correlation to the downstream robotic
  manipulation tasks (i.e., manipulation centricity). Interestingly, we find that
  the "manipulation centricity" is a strong indicator of success rates when
  applied to downstream tasks. Drawing from these findings, we propose
  Manipulation Centric Representation (MCR), a foundation representation learning
  framework capturing both visual features and the dynamics information such as
  actions and proprioceptions of manipulation tasks to improve manipulation
  centricity. Specifically, we pre-train a visual encoder on the DROID robotic
  dataset and leverage motion-relevant data such as robot proprioceptive states
  and actions. We introduce a novel contrastive loss that aligns visual
  observations with the robot's proprioceptive state-action dynamics, combined
  with a behavior cloning (BC)-like actor loss to predict actions during
  pre-training, along with a time contrastive loss. Empirical results across 4
  simulation domains with 20 tasks verify that MCR outperforms the strongest
  baseline method by 14.8%. Moreover, MCR boosts the performance of
  data-efficient learning with a UR5e arm on 3 real-world tasks by 76.9%. Project
  website: https://robots-pretrain-robots.github.io/.
paper_ai_summary: A new representation learning framework, MCR, enhances manipulation-centric visual representations by integrating both visual features and dynamic information, leading to improved performance in robotic manipulation tasks.
paper_ai_keywords: manipulation centricity Manipulation Centric Representation (MCR) foundation representation learning visual encoder proprioceptive states actions contrastive loss actor loss time contrastive loss simulation domains UR5e arm