title: RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation
summary: The ability to leverage heterogeneous robotic experience from different
  robots and tasks to quickly master novel skills and embodiments has the
  potential to transform robot learning. Inspired by recent advances in
  foundation models for vision and language, we propose a foundation agent for
  robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned
  decision transformer capable of consuming multi-embodiment action-labelled
  visual experience. This data spans a large repertoire of motor control skills
  from simulated and real robotic arms with varying sets of observations and
  actions. With RoboCat, we demonstrate the ability to generalise to new tasks
  and robots, both zero-shot as well as through adaptation using only 100--1000
  examples for the target task. We also show how a trained model itself can be
  used to generate data for subsequent training iterations, thus providing a
  basic building block for an autonomous improvement loop. We investigate the
  agent's capabilities, with large-scale evaluations both in simulation and on
  three different real robot embodiments. We find that as we grow and diversify
  its training data, RoboCat not only shows signs of cross-task transfer, but
  also becomes more efficient at adapting to new tasks.
paper_authors._id: 649298bd89715cc48a94dd7e
paper_authors.name: Konstantinos Bousmalis
paper_authors._id: 649298bd89715cc48a94dd7f
paper_authors.name: Giulia Vezzani
paper_authors._id: 649298bd89715cc48a94dd80
paper_authors.name: Dushyant Rao
paper_authors._id: 649298bd89715cc48a94dd81
paper_authors.name: Coline Devin
paper_authors._id: 649298bd89715cc48a94dd82
paper_authors.name: Alex X. Lee
paper_authors._id: 649298bd89715cc48a94dd83
paper_authors.name: Maria Bauza
paper_authors._id: 649298bd89715cc48a94dd84
paper_authors.name: Todor Davchev
paper_authors._id: 649298bd89715cc48a94dd85
paper_authors.name: Yuxiang Zhou
paper_authors._id: 649298bd89715cc48a94dd86
paper_authors.name: Agrim Gupta
paper_authors._id: 649298bd89715cc48a94dd87
paper_authors.name: Akhil Raju
paper_authors._id: 649298bd89715cc48a94dd88
paper_authors.name: Antoine Laurens
paper_authors._id: 649298bd89715cc48a94dd89
paper_authors.name: Claudio Fantacci
paper_authors._id: 649298bd89715cc48a94dd8a
paper_authors.name: Valentin Dalibard
paper_authors._id: 649298bd89715cc48a94dd8b
paper_authors.name: Martina Zambelli
paper_authors._id: 649298bd89715cc48a94dd8c
paper_authors.name: Murilo Martins
paper_authors._id: 649298bd89715cc48a94dd8d
paper_authors.name: Rugile Pevceviciute
paper_authors._id: 649298bd89715cc48a94dd8e
paper_authors.name: Michiel Blokzijl
paper_authors._id: 649298bd89715cc48a94dd8f
paper_authors.name: Misha Denil
paper_authors._id: 649298bd89715cc48a94dd90
paper_authors.name: Nathan Batchelor
paper_authors._id: 649298bd89715cc48a94dd91
paper_authors.name: Thomas Lampe
paper_authors._id: 649298bd89715cc48a94dd92
paper_authors.name: Emilio Parisotto
paper_authors._id: 649298bd89715cc48a94dd93
paper_authors.name: Konrad Żołna
paper_authors._id: 649298bd89715cc48a94dd94
paper_authors.name: Scott Reed
paper_authors._id: 649298bd89715cc48a94dd95
paper_authors.name: Sergio Gómez Colmenarejo
paper_authors._id: 649298bd89715cc48a94dd96
paper_authors.name: Jon Scholz
paper_authors._id: 649298bd89715cc48a94dd97
paper_authors.name: Abbas Abdolmaleki
paper_authors._id: 649298bd89715cc48a94dd98
paper_authors.name: Oliver Groth
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-22T08:52:54.465Z
paper_authors._id: 6372a44e0f3db8b7e625f6dc
paper_authors.avatarUrl: /avatars/814237f0198f69407a2981ee01abb5bc.svg
paper_authors.fullname: Oliver Groth
paper_authors.type: user
paper_authors.user: ogroth
paper_authors._id: 649298bd89715cc48a94dd99
paper_authors.name: Jean-Baptiste Regli
paper_authors._id: 649298bd89715cc48a94dd9a
paper_authors.name: Oleg Sushkov
paper_authors._id: 649298bd89715cc48a94dd9b
paper_authors.name: Tom Rothörl
paper_authors._id: 649298bd89715cc48a94dd9c
paper_authors.name: José Enrique Chen
paper_authors._id: 649298bd89715cc48a94dd9d
paper_authors.name: Yusuf Aytar
paper_authors._id: 649298bd89715cc48a94dd9e
paper_authors.name: Dave Barker
paper_authors._id: 649298bd89715cc48a94dd9f
paper_authors.name: Joy Ortiz
paper_authors._id: 649298bd89715cc48a94dda0
paper_authors.name: Martin Riedmiller
paper_authors._id: 649298bd89715cc48a94dda1
paper_authors.name: Jost Tobias Springenberg
paper_authors._id: 649298bd89715cc48a94dda2
paper_authors.name: Raia Hadsell
paper_authors._id: 649298bd89715cc48a94dda3
paper_authors.name: Francesco Nori
paper_authors._id: 649298bd89715cc48a94dda4
paper_authors.name: Nicolas Heess
paper_title: RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation
paper_summary: The ability to leverage heterogeneous robotic experience from different
  robots and tasks to quickly master novel skills and embodiments has the
  potential to transform robot learning. Inspired by recent advances in
  foundation models for vision and language, we propose a foundation agent for
  robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned
  decision transformer capable of consuming multi-embodiment action-labelled
  visual experience. This data spans a large repertoire of motor control skills
  from simulated and real robotic arms with varying sets of observations and
  actions. With RoboCat, we demonstrate the ability to generalise to new tasks
  and robots, both zero-shot as well as through adaptation using only 100--1000
  examples for the target task. We also show how a trained model itself can be
  used to generate data for subsequent training iterations, thus providing a
  basic building block for an autonomous improvement loop. We investigate the
  agent's capabilities, with large-scale evaluations both in simulation and on
  three different real robot embodiments. We find that as we grow and diversify
  its training data, RoboCat not only shows signs of cross-task transfer, but
  also becomes more efficient at adapting to new tasks.
paper_ai_summary: RoboCat, a visual goal-conditioned decision transformer, leverages heterogeneous robotic experience to achieve zero-shot and few-shot adaptation across various tasks and robot embodiments.
paper_ai_keywords: foundation agent decision transformer visual goal-conditioned multi-embodiment action-labelled motor control skills zero-shot few-shot adaptation cross-task transfer