title: Robot Learning with Sensorimotor Pre-training
summary: We present a self-supervised sensorimotor pre-training approach for robotics.
  Our model, called RPT, is a Transformer that operates on sequences of
  sensorimotor tokens. Given a sequence of camera images, proprioceptive robot
  states, and past actions, we encode the interleaved sequence into tokens, mask
  out a random subset, and train a model to predict the masked-out content. We
  hypothesize that if the robot can predict the missing content it has acquired a
  good model of the physical world that can enable it to act. RPT is designed to
  operate on latent visual representations which makes prediction tractable,
  enables scaling to 10x larger models, and 10 Hz inference on a real robot. To
  evaluate our approach, we collect a dataset of 20,000 real-world trajectories
  over 9 months using a combination of motion planning and model-based grasping
  algorithms. We find that pre-training on this data consistently outperforms
  training from scratch, leads to 2x improvements in the block stacking task, and
  has favorable scaling properties.
paper_authors._id: 648fd8841ba574b69dfa0e10
paper_authors.name: Ilija Radosavovic
paper_authors._id: 648fd8841ba574b69dfa0e11
paper_authors.name: Baifeng Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-19T11:44:38.643Z
paper_authors._id: 649004218f7cbbc94c782db6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
paper_authors.fullname: Baifeng Shi
paper_authors.type: user
paper_authors.user: bfshi
paper_authors._id: 648fd8841ba574b69dfa0e12
paper_authors.name: Letian Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-19T19:44:31.589Z
paper_authors._id: 64908d6b0c18343a09361b93
paper_authors.avatarUrl: /avatars/2fa44344fe760f68f19de775b76f45b8.svg
paper_authors.fullname: Max (Letian) Fu
paper_authors.type: user
paper_authors.user: mlfu7
paper_authors._id: 648fd8841ba574b69dfa0e13
paper_authors.name: Ken Goldberg
paper_authors._id: 648fd8841ba574b69dfa0e14
paper_authors.name: Trevor Darrell
paper_authors._id: 648fd8841ba574b69dfa0e15
paper_authors.name: Jitendra Malik
paper_title: Robot Learning with Sensorimotor Pre-training
paper_summary: We present a self-supervised sensorimotor pre-training approach for robotics.
  Our model, called RPT, is a Transformer that operates on sequences of
  sensorimotor tokens. Given a sequence of camera images, proprioceptive robot
  states, and past actions, we encode the interleaved sequence into tokens, mask
  out a random subset, and train a model to predict the masked-out content. We
  hypothesize that if the robot can predict the missing content it has acquired a
  good model of the physical world that can enable it to act. RPT is designed to
  operate on latent visual representations which makes prediction tractable,
  enables scaling to 10x larger models, and 10 Hz inference on a real robot. To
  evaluate our approach, we collect a dataset of 20,000 real-world trajectories
  over 9 months using a combination of motion planning and model-based grasping
  algorithms. We find that pre-training on this data consistently outperforms
  training from scratch, leads to 2x improvements in the block stacking task, and
  has favorable scaling properties.
paper_ai_summary: A self-supervised sensorimotor pre-training method using a Transformer model on sequences of sensorimotor tokens improves robotic performance in tasks like block stacking.
paper_ai_keywords: Transformer sensorimotor tokens camera images proprioceptive states past actions latent visual representations