RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation
The ability to leverage heterogeneous robotic experience from different
robots and tasks to quickly master novel skills and embodiments has the
potential to transform robot learning. Inspired by recent advances in
foundation models for vision and language, we propose a foundation agent for
robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned
decision transformer capable of consuming multi-embodiment action-labelled
visual experience. This data spans a large repertoire of motor control skills
from simulated and real robotic arms with varying sets of observations and
actions. With RoboCat, we demonstrate the ability to generalise to new tasks
and robots, both zero-shot as well as through adaptation using only 100--1000
examples for the target task. We also show how a trained model itself can be
used to generate data for subsequent training iterations, thus providing a
basic building block for an autonomous improvement loop. We investigate the
agent's capabilities, with large-scale evaluations both in simulation and on
three different real robot embodiments. We find that as we grow and diversify
its training data, RoboCat not only shows signs of cross-task transfer, but
also becomes more efficient at adapting to new tasks.
649298bd89715cc48a94dd7e
Konstantinos Bousmalis
649298bd89715cc48a94dd7f
Giulia Vezzani
649298bd89715cc48a94dd80
Dushyant Rao
649298bd89715cc48a94dd81
Coline Devin
649298bd89715cc48a94dd82
Alex X. Lee
649298bd89715cc48a94dd83
Maria Bauza
649298bd89715cc48a94dd84
Todor Davchev
649298bd89715cc48a94dd85
Yuxiang Zhou
649298bd89715cc48a94dd86
Agrim Gupta
649298bd89715cc48a94dd87
Akhil Raju
649298bd89715cc48a94dd88
Antoine Laurens
649298bd89715cc48a94dd89
Claudio Fantacci
649298bd89715cc48a94dd8a
Valentin Dalibard
649298bd89715cc48a94dd8b
Martina Zambelli
649298bd89715cc48a94dd8c
Murilo Martins
649298bd89715cc48a94dd8d
Rugile Pevceviciute
649298bd89715cc48a94dd8e
Michiel Blokzijl
649298bd89715cc48a94dd8f
Misha Denil
649298bd89715cc48a94dd90
Nathan Batchelor
649298bd89715cc48a94dd91
Thomas Lampe
649298bd89715cc48a94dd92
Emilio Parisotto
649298bd89715cc48a94dd93
Konrad Żołna
649298bd89715cc48a94dd94
Scott Reed
649298bd89715cc48a94dd95
Sergio Gómez Colmenarejo
649298bd89715cc48a94dd96
Jon Scholz
649298bd89715cc48a94dd97
Abbas Abdolmaleki
649298bd89715cc48a94dd98
Oliver Groth
admin_assigned
2023-06-22T08:52:54.465Z
6372a44e0f3db8b7e625f6dc
/avatars/814237f0198f69407a2981ee01abb5bc.svg
Oliver Groth
user
ogroth
649298bd89715cc48a94dd99
Jean-Baptiste Regli
649298bd89715cc48a94dd9a
Oleg Sushkov
649298bd89715cc48a94dd9b
Tom Rothörl
649298bd89715cc48a94dd9c
José Enrique Chen
649298bd89715cc48a94dd9d
Yusuf Aytar
649298bd89715cc48a94dd9e
Dave Barker
649298bd89715cc48a94dd9f
Joy Ortiz
649298bd89715cc48a94dda0
Martin Riedmiller
649298bd89715cc48a94dda1
Jost Tobias Springenberg
649298bd89715cc48a94dda2
Raia Hadsell
649298bd89715cc48a94dda3
Francesco Nori
649298bd89715cc48a94dda4
Nicolas Heess
RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation
The ability to leverage heterogeneous robotic experience from different
robots and tasks to quickly master novel skills and embodiments has the
potential to transform robot learning. Inspired by recent advances in
foundation models for vision and language, we propose a foundation agent for
robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned
decision transformer capable of consuming multi-embodiment action-labelled
visual experience. This data spans a large repertoire of motor control skills
from simulated and real robotic arms with varying sets of observations and
actions. With RoboCat, we demonstrate the ability to generalise to new tasks
and robots, both zero-shot as well as through adaptation using only 100--1000
examples for the target task. We also show how a trained model itself can be
used to generate data for subsequent training iterations, thus providing a
basic building block for an autonomous improvement loop. We investigate the
agent's capabilities, with large-scale evaluations both in simulation and on
three different real robot embodiments. We find that as we grow and diversify
its training data, RoboCat not only shows signs of cross-task transfer, but
also becomes more efficient at adapting to new tasks.
RoboCat, a visual goal-conditioned decision transformer, leverages heterogeneous robotic experience to achieve zero-shot and few-shot adaptation across various tasks and robot embodiments.
foundation agent decision transformer visual goal-conditioned multi-embodiment action-labelled motor control skills zero-shot few-shot adaptation cross-task transfer