title: RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic
    Control
summary: We study how vision-language models trained on Internet-scale data can be
  incorporated directly into end-to-end robotic control to boost generalization
  and enable emergent semantic reasoning. Our goal is to enable a single
  end-to-end trained model to both learn to map robot observations to actions and
  enjoy the benefits of large-scale pretraining on language and vision-language
  data from the web. To this end, we propose to co-fine-tune state-of-the-art
  vision-language models on both robotic trajectory data and Internet-scale
  vision-language tasks, such as visual question answering. In contrast to other
  approaches, we propose a simple, general recipe to achieve this goal: in order
  to fit both natural language responses and robotic actions into the same
  format, we express the actions as text tokens and incorporate them directly
  into the training set of the model in the same way as natural language tokens.
  We refer to such category of models as vision-language-action models (VLA) and
  instantiate an example of such a model, which we call RT-2. Our extensive
  evaluation (6k evaluation trials) shows that our approach leads to performant
  robotic policies and enables RT-2 to obtain a range of emergent capabilities
  from Internet-scale training. This includes significantly improved
  generalization to novel objects, the ability to interpret commands not present
  in the robot training data (such as placing an object onto a particular number
  or icon), and the ability to perform rudimentary reasoning in response to user
  commands (such as picking up the smallest or largest object, or the one closest
  to another object). We further show that incorporating chain of thought
  reasoning allows RT-2 to perform multi-stage semantic reasoning, for example
  figuring out which object to pick up for use as an improvised hammer (a rock),
  or which type of drink is best suited for someone who is tired (an energy
  drink).
paper_authors._id: 64c8661dc864d962edee7248
paper_authors.name: Anthony Brohan
paper_authors._id: 64c8661dc864d962edee7249
paper_authors.name: Noah Brown
paper_authors._id: 64c8661dc864d962edee724a
paper_authors.name: Justice Carbajal
paper_authors._id: 64c8661dc864d962edee724b
paper_authors.name: Yevgen Chebotar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T18:23:19.563Z
paper_authors._id: 64c93f9c9471c54a9438162c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c93f9c9471c54a9438162c/bYGUqFsF67VuAuXecjxjX.jpeg
paper_authors.fullname: Yevgen Chebotar
paper_authors.type: user
paper_authors.user: ychebotar
paper_authors._id: 64c8661dc864d962edee724c
paper_authors.name: Xi Chen
paper_authors._id: 64c8661dc864d962edee724d
paper_authors.name: Krzysztof Choromanski
paper_authors._id: 64c8661dc864d962edee724e
paper_authors.name: Tianli Ding
paper_authors._id: 64c8661dc864d962edee724f
paper_authors.name: Danny Driess
paper_authors._id: 64c8661dc864d962edee7250
paper_authors.name: Avinava Dubey
paper_authors._id: 64c8661dc864d962edee7251
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T07:55:34.448Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_authors._id: 64c8661dc864d962edee7252
paper_authors.name: Pete Florence
paper_authors._id: 64c8661dc864d962edee7253
paper_authors.name: Chuyuan Fu
paper_authors._id: 64c8661dc864d962edee7254
paper_authors.name: Montse Gonzalez Arenas
paper_authors._id: 64c8661dc864d962edee7255
paper_authors.name: Keerthana Gopalakrishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T07:55:57.663Z
paper_authors._id: 632f7dee34c562cb1d380fdc
paper_authors.avatarUrl: /avatars/d689e260535b8aa775654160028d4ec8.svg
paper_authors.fullname: Keerthana Gopalakrishnan
paper_authors.type: user
paper_authors.user: keerthanpg
paper_authors._id: 64c8661dc864d962edee7256
paper_authors.name: Kehang Han
paper_authors._id: 64c8661dc864d962edee7257
paper_authors.name: Karol Hausman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T18:23:23.271Z
paper_authors._id: 64c93e623a3f428da67d9310
paper_authors.avatarUrl: /avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
paper_authors.fullname: Karol Hausman
paper_authors.type: user
paper_authors.user: KarolHausman
paper_authors._id: 64c8661dc864d962edee7258
paper_authors.name: Alexander Herzog
paper_authors._id: 64c8661dc864d962edee7259
paper_authors.name: Jasmine Hsu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T07:56:38.829Z
paper_authors._id: 646c7d0a93badbc8c2e7593c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/8rCfhERYEuImofjSSDmQS.jpeg
paper_authors.fullname: Jasmine Hsu
paper_authors.type: user
paper_authors.user: Jasmine2023
paper_authors._id: 64c8661dc864d962edee725a
paper_authors.name: Brian Ichter
paper_authors._id: 64c8661dc864d962edee725b
paper_authors.name: Alex Irpan
paper_authors._id: 64c8661dc864d962edee725c
paper_authors.name: Nikhil Joshi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T07:57:03.540Z
paper_authors._id: 63c15485894342c8964c481d
paper_authors.avatarUrl: /avatars/cd938a12eb73a8395872f4058d85b0c5.svg
paper_authors.fullname: nikhil joshi
paper_authors.type: user
paper_authors.user: valeska
paper_authors._id: 64c8661dc864d962edee725d
paper_authors.name: Ryan Julian
paper_authors._id: 64c8661dc864d962edee725e
paper_authors.name: Dmitry Kalashnikov
paper_authors._id: 64c8661dc864d962edee725f
paper_authors.name: Yuheng Kuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T07:57:49.633Z
paper_authors._id: 6312dd4f95407887cb7c9cd7
paper_authors.avatarUrl: /avatars/44a0e80ca9bbd2138eecc6b2a8b8b226.svg
paper_authors.fullname: Yuheng Kuang
paper_authors.type: user
paper_authors.user: kuangyh
paper_authors._id: 64c8661dc864d962edee7260
paper_authors.name: Isabel Leal
paper_authors._id: 64c8661dc864d962edee7261
paper_authors.name: Lisa Lee
paper_authors._id: 64c8661dc864d962edee7262
paper_authors.name: Tsang-Wei Edward Lee
paper_authors._id: 64c8661dc864d962edee7263
paper_authors.name: Sergey Levine
paper_authors._id: 64c8661dc864d962edee7264
paper_authors.name: Yao Lu
paper_authors._id: 64c8661dc864d962edee7265
paper_authors.name: Henryk Michalewski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T07:59:09.705Z
paper_authors._id: 6343ead529de2293e7e01889
paper_authors.avatarUrl: /avatars/dc141ec176bf8b23306be64dbb5b22d1.svg
paper_authors.fullname: Henryk Michalewski
paper_authors.type: user
paper_authors.user: henrykmichalewski
paper_authors._id: 64c8661dc864d962edee7266
paper_authors.name: Igor Mordatch
paper_authors._id: 64c8661dc864d962edee7267
paper_authors.name: Karl Pertsch
paper_authors._id: 64c8661dc864d962edee7268
paper_authors.name: Kanishka Rao
paper_authors._id: 64c8661dc864d962edee7269
paper_authors.name: Krista Reymann
paper_authors._id: 64c8661dc864d962edee726a
paper_authors.name: Michael Ryoo
paper_authors._id: 64c8661dc864d962edee726b
paper_authors.name: Grecia Salazar
paper_authors._id: 64c8661dc864d962edee726c
paper_authors.name: Pannag Sanketi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:49:19.264Z
paper_authors._id: 66b56f73fc66a466e6104bfb
paper_authors.avatarUrl: /avatars/74855aaf580144fafead4b93d6a24a46.svg
paper_authors.fullname: Pannag Sanketi
paper_authors.type: user
paper_authors.user: prsanky
paper_authors._id: 64c8661dc864d962edee726d
paper_authors.name: Pierre Sermanet
paper_authors._id: 64c8661dc864d962edee726e
paper_authors.name: Jaspiar Singh
paper_authors._id: 64c8661dc864d962edee726f
paper_authors.name: Anikait Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:15:18.300Z
paper_authors._id: 6511ee845b7e52b0251fdee9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6511ee845b7e52b0251fdee9/hTIwiIYBGOVnIrxtpri83.png
paper_authors.fullname: Anikait Singh
paper_authors.type: user
paper_authors.user: Asap7772
paper_authors._id: 64c8661dc864d962edee7270
paper_authors.name: Radu Soricut
paper_authors._id: 64c8661dc864d962edee7271
paper_authors.name: Huong Tran
paper_authors.statusLastChangedAt: 2023-08-01T09:06:12.977Z
paper_authors._id: 64c8661dc864d962edee7272
paper_authors.name: Vincent Vanhoucke
paper_authors._id: 64c8661dc864d962edee7273
paper_authors.name: Quan Vuong
paper_authors._id: 64c8661dc864d962edee7274
paper_authors.name: Ayzaan Wahid
paper_authors._id: 64c8661dc864d962edee7275
paper_authors.name: Stefan Welker
paper_authors._id: 64c8661dc864d962edee7276
paper_authors.name: Paul Wohlhart
paper_authors._id: 64c8661dc864d962edee7277
paper_authors.name: Jialin Wu
paper_authors._id: 64c8661dc864d962edee7278
paper_authors.name: Fei Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T18:23:15.425Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_authors._id: 64c8661dc864d962edee7279
paper_authors.name: Ted Xiao
paper_authors._id: 64c8661dc864d962edee727a
paper_authors.name: Peng Xu
paper_authors._id: 64c8661dc864d962edee727b
paper_authors.name: Sichun Xu
paper_authors._id: 64c8661dc864d962edee727c
paper_authors.name: Tianhe Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T19:36:26.071Z
paper_authors._id: 64c95527d33e6a7f82b8b911
paper_authors.avatarUrl: /avatars/5efcf5d5a84766bbd3a5ec2354233f1c.svg
paper_authors.fullname: Tianhe Yu
paper_authors.type: user
paper_authors.user: keviny66
paper_authors._id: 64c8661dc864d962edee727d
paper_authors.name: Brianna Zitkovich
paper_title: RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic
    Control
paper_summary: We study how vision-language models trained on Internet-scale data can be
  incorporated directly into end-to-end robotic control to boost generalization
  and enable emergent semantic reasoning. Our goal is to enable a single
  end-to-end trained model to both learn to map robot observations to actions and
  enjoy the benefits of large-scale pretraining on language and vision-language
  data from the web. To this end, we propose to co-fine-tune state-of-the-art
  vision-language models on both robotic trajectory data and Internet-scale
  vision-language tasks, such as visual question answering. In contrast to other
  approaches, we propose a simple, general recipe to achieve this goal: in order
  to fit both natural language responses and robotic actions into the same
  format, we express the actions as text tokens and incorporate them directly
  into the training set of the model in the same way as natural language tokens.
  We refer to such category of models as vision-language-action models (VLA) and
  instantiate an example of such a model, which we call RT-2. Our extensive
  evaluation (6k evaluation trials) shows that our approach leads to performant
  robotic policies and enables RT-2 to obtain a range of emergent capabilities
  from Internet-scale training. This includes significantly improved
  generalization to novel objects, the ability to interpret commands not present
  in the robot training data (such as placing an object onto a particular number
  or icon), and the ability to perform rudimentary reasoning in response to user
  commands (such as picking up the smallest or largest object, or the one closest
  to another object). We further show that incorporating chain of thought
  reasoning allows RT-2 to perform multi-stage semantic reasoning, for example
  figuring out which object to pick up for use as an improvised hammer (a rock),
  or which type of drink is best suited for someone who is tired (an energy
  drink).
paper_ai_summary: Vision-language-action models, trained on large-scale vision and language data, are integrated into robotic control to enhance generalization and semantic reasoning.
paper_ai_keywords: vision-language models end-to-end robotic control co-fine-tuning robotic trajectory data vision-language tasks visual question answering text tokens vision-language-action models VLA RT-2 emergent capabilities chain of thought reasoning