RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic
  Control
We study how vision-language models trained on Internet-scale data can be
incorporated directly into end-to-end robotic control to boost generalization
and enable emergent semantic reasoning. Our goal is to enable a single
end-to-end trained model to both learn to map robot observations to actions and
enjoy the benefits of large-scale pretraining on language and vision-language
data from the web. To this end, we propose to co-fine-tune state-of-the-art
vision-language models on both robotic trajectory data and Internet-scale
vision-language tasks, such as visual question answering. In contrast to other
approaches, we propose a simple, general recipe to achieve this goal: in order
to fit both natural language responses and robotic actions into the same
format, we express the actions as text tokens and incorporate them directly
into the training set of the model in the same way as natural language tokens.
We refer to such category of models as vision-language-action models (VLA) and
instantiate an example of such a model, which we call RT-2. Our extensive
evaluation (6k evaluation trials) shows that our approach leads to performant
robotic policies and enables RT-2 to obtain a range of emergent capabilities
from Internet-scale training. This includes significantly improved
generalization to novel objects, the ability to interpret commands not present
in the robot training data (such as placing an object onto a particular number
or icon), and the ability to perform rudimentary reasoning in response to user
commands (such as picking up the smallest or largest object, or the one closest
to another object). We further show that incorporating chain of thought
reasoning allows RT-2 to perform multi-stage semantic reasoning, for example
figuring out which object to pick up for use as an improvised hammer (a rock),
or which type of drink is best suited for someone who is tired (an energy
drink).
64c8661dc864d962edee7248
Anthony Brohan
64c8661dc864d962edee7249
Noah Brown
64c8661dc864d962edee724a
Justice Carbajal
64c8661dc864d962edee724b
Yevgen Chebotar
claimed_verified
2023-08-01T18:23:19.563Z
64c93f9c9471c54a9438162c
https://cdn-avatars.huggingface.co/v1/production/uploads/64c93f9c9471c54a9438162c/bYGUqFsF67VuAuXecjxjX.jpeg
Yevgen Chebotar
user
ychebotar
64c8661dc864d962edee724c
Xi Chen
64c8661dc864d962edee724d
Krzysztof Choromanski
64c8661dc864d962edee724e
Tianli Ding
64c8661dc864d962edee724f
Danny Driess
64c8661dc864d962edee7250
Avinava Dubey
64c8661dc864d962edee7251
Chelsea Finn
admin_assigned
2023-08-01T07:55:34.448Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
64c8661dc864d962edee7252
Pete Florence
64c8661dc864d962edee7253
Chuyuan Fu
64c8661dc864d962edee7254
Montse Gonzalez Arenas
64c8661dc864d962edee7255
Keerthana Gopalakrishnan
admin_assigned
2023-08-01T07:55:57.663Z
632f7dee34c562cb1d380fdc
/avatars/d689e260535b8aa775654160028d4ec8.svg
Keerthana Gopalakrishnan
user
keerthanpg
64c8661dc864d962edee7256
Kehang Han
64c8661dc864d962edee7257
Karol Hausman
claimed_verified
2023-08-01T18:23:23.271Z
64c93e623a3f428da67d9310
/avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
Karol Hausman
user
KarolHausman
64c8661dc864d962edee7258
Alexander Herzog
64c8661dc864d962edee7259
Jasmine Hsu
admin_assigned
2023-08-01T07:56:38.829Z
646c7d0a93badbc8c2e7593c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/8rCfhERYEuImofjSSDmQS.jpeg
Jasmine Hsu
user
Jasmine2023
64c8661dc864d962edee725a
Brian Ichter
64c8661dc864d962edee725b
Alex Irpan
64c8661dc864d962edee725c
Nikhil Joshi
admin_assigned
2023-08-01T07:57:03.540Z
63c15485894342c8964c481d
/avatars/cd938a12eb73a8395872f4058d85b0c5.svg
nikhil joshi
user
valeska
64c8661dc864d962edee725d
Ryan Julian
64c8661dc864d962edee725e
Dmitry Kalashnikov
64c8661dc864d962edee725f
Yuheng Kuang
admin_assigned
2023-08-01T07:57:49.633Z
6312dd4f95407887cb7c9cd7
/avatars/44a0e80ca9bbd2138eecc6b2a8b8b226.svg
Yuheng Kuang
user
kuangyh
64c8661dc864d962edee7260
Isabel Leal
64c8661dc864d962edee7261
Lisa Lee
64c8661dc864d962edee7262
Tsang-Wei Edward Lee
64c8661dc864d962edee7263
Sergey Levine
64c8661dc864d962edee7264
Yao Lu
64c8661dc864d962edee7265
Henryk Michalewski
admin_assigned
2023-08-01T07:59:09.705Z
6343ead529de2293e7e01889
/avatars/dc141ec176bf8b23306be64dbb5b22d1.svg
Henryk Michalewski
user
henrykmichalewski
64c8661dc864d962edee7266
Igor Mordatch
64c8661dc864d962edee7267
Karl Pertsch
64c8661dc864d962edee7268
Kanishka Rao
64c8661dc864d962edee7269
Krista Reymann
64c8661dc864d962edee726a
Michael Ryoo
64c8661dc864d962edee726b
Grecia Salazar
64c8661dc864d962edee726c
Pannag Sanketi
claimed_verified
2024-08-09T07:49:19.264Z
66b56f73fc66a466e6104bfb
/avatars/74855aaf580144fafead4b93d6a24a46.svg
Pannag Sanketi
user
prsanky
64c8661dc864d962edee726d
Pierre Sermanet
64c8661dc864d962edee726e
Jaspiar Singh
64c8661dc864d962edee726f
Anikait Singh
claimed_verified
2024-08-19T22:15:18.300Z
6511ee845b7e52b0251fdee9
https://cdn-avatars.huggingface.co/v1/production/uploads/6511ee845b7e52b0251fdee9/hTIwiIYBGOVnIrxtpri83.png
Anikait Singh
user
Asap7772
64c8661dc864d962edee7270
Radu Soricut
64c8661dc864d962edee7271
Huong Tran
2023-08-01T09:06:12.977Z
64c8661dc864d962edee7272
Vincent Vanhoucke
64c8661dc864d962edee7273
Quan Vuong
64c8661dc864d962edee7274
Ayzaan Wahid
64c8661dc864d962edee7275
Stefan Welker
64c8661dc864d962edee7276
Paul Wohlhart
64c8661dc864d962edee7277
Jialin Wu
64c8661dc864d962edee7278
Fei Xia
claimed_verified
2023-08-01T18:23:15.425Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
64c8661dc864d962edee7279
Ted Xiao
64c8661dc864d962edee727a
Peng Xu
64c8661dc864d962edee727b
Sichun Xu
64c8661dc864d962edee727c
Tianhe Yu
claimed_verified
2023-08-01T19:36:26.071Z
64c95527d33e6a7f82b8b911
/avatars/5efcf5d5a84766bbd3a5ec2354233f1c.svg
Tianhe Yu
user
keviny66
64c8661dc864d962edee727d
Brianna Zitkovich
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic
  Control
We study how vision-language models trained on Internet-scale data can be
incorporated directly into end-to-end robotic control to boost generalization
and enable emergent semantic reasoning. Our goal is to enable a single
end-to-end trained model to both learn to map robot observations to actions and
enjoy the benefits of large-scale pretraining on language and vision-language
data from the web. To this end, we propose to co-fine-tune state-of-the-art
vision-language models on both robotic trajectory data and Internet-scale
vision-language tasks, such as visual question answering. In contrast to other
approaches, we propose a simple, general recipe to achieve this goal: in order
to fit both natural language responses and robotic actions into the same
format, we express the actions as text tokens and incorporate them directly
into the training set of the model in the same way as natural language tokens.
We refer to such category of models as vision-language-action models (VLA) and
instantiate an example of such a model, which we call RT-2. Our extensive
evaluation (6k evaluation trials) shows that our approach leads to performant
robotic policies and enables RT-2 to obtain a range of emergent capabilities
from Internet-scale training. This includes significantly improved
generalization to novel objects, the ability to interpret commands not present
in the robot training data (such as placing an object onto a particular number
or icon), and the ability to perform rudimentary reasoning in response to user
commands (such as picking up the smallest or largest object, or the one closest
to another object). We further show that incorporating chain of thought
reasoning allows RT-2 to perform multi-stage semantic reasoning, for example
figuring out which object to pick up for use as an improvised hammer (a rock),
or which type of drink is best suited for someone who is tired (an energy
drink).
Vision-language-action models, trained on large-scale vision and language data, are integrated into robotic control to enhance generalization and semantic reasoning.
vision-language models end-to-end robotic control co-fine-tuning robotic trajectory data vision-language tasks visual question answering text tokens vision-language-action models VLA RT-2 emergent capabilities chain of thought reasoning