title: AutoRT: Embodied Foundation Models for Large Scale Orchestration of
    Robotic Agents
summary: Foundation models that incorporate language, vision, and more recently
  actions have revolutionized the ability to harness internet scale data to
  reason about useful tasks. However, one of the key challenges of training
  embodied foundation models is the lack of data grounded in the physical world.
  In this paper, we propose AutoRT, a system that leverages existing foundation
  models to scale up the deployment of operational robots in completely unseen
  scenarios with minimal human supervision. AutoRT leverages vision-language
  models (VLMs) for scene understanding and grounding, and further uses large
  language models (LLMs) for proposing diverse and novel instructions to be
  performed by a fleet of robots. Guiding data collection by tapping into the
  knowledge of foundation models enables AutoRT to effectively reason about
  autonomy tradeoffs and safety while significantly scaling up data collection
  for robot learning. We demonstrate AutoRT proposing instructions to over 20
  robots across multiple buildings and collecting 77k real robot episodes via
  both teleoperation and autonomous robot policies. We experimentally show that
  such "in-the-wild" data collected by AutoRT is significantly more diverse, and
  that AutoRT's use of LLMs allows for instruction following data collection
  robots that can align to human preferences.
paper_authors._id: 65b0828cc11502dcf18b86f6
paper_authors.name: Michael Ahn
paper_authors._id: 65b0828cc11502dcf18b86f7
paper_authors.name: Debidatta Dwibedi
paper_authors._id: 65b0828cc11502dcf18b86f8
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:05:34.239Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_authors._id: 65b0828cc11502dcf18b86f9
paper_authors.name: Montse Gonzalez Arenas
paper_authors._id: 65b0828cc11502dcf18b86fa
paper_authors.name: Keerthana Gopalakrishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:05:51.318Z
paper_authors._id: 632f7dee34c562cb1d380fdc
paper_authors.avatarUrl: /avatars/d689e260535b8aa775654160028d4ec8.svg
paper_authors.fullname: Keerthana Gopalakrishnan
paper_authors.type: user
paper_authors.user: keerthanpg
paper_authors._id: 65b0828cc11502dcf18b86fb
paper_authors.name: Karol Hausman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:05:59.131Z
paper_authors._id: 64c93e623a3f428da67d9310
paper_authors.avatarUrl: /avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
paper_authors.fullname: Karol Hausman
paper_authors.type: user
paper_authors.user: KarolHausman
paper_authors._id: 65b0828cc11502dcf18b86fc
paper_authors.name: Brian Ichter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:06:08.348Z
paper_authors._id: 633a26ab474cfeb1a864dc56
paper_authors.avatarUrl: /avatars/cd40accd894fc78810fc0d5108f413e9.svg
paper_authors.fullname: Brian I
paper_authors.type: user
paper_authors.user: brianichter
paper_authors._id: 65b0828cc11502dcf18b86fd
paper_authors.name: Alex Irpan
paper_authors._id: 65b0828cc11502dcf18b86fe
paper_authors.name: Nikhil Joshi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:06:47.545Z
paper_authors._id: 63c15485894342c8964c481d
paper_authors.avatarUrl: /avatars/cd938a12eb73a8395872f4058d85b0c5.svg
paper_authors.fullname: nikhil joshi
paper_authors.type: user
paper_authors.user: valeska
paper_authors._id: 65b0828cc11502dcf18b86ff
paper_authors.name: Ryan Julian
paper_authors._id: 65b0828cc11502dcf18b8700
paper_authors.name: Sean Kirmani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:06:59.532Z
paper_authors._id: 648c85f33ffc11989bddcce1
paper_authors.avatarUrl: /avatars/bdde73b95a36e32cb2975656cca46022.svg
paper_authors.fullname: Sean Kirmani
paper_authors.type: user
paper_authors.user: skirmani
paper_authors._id: 65b0828cc11502dcf18b8701
paper_authors.name: Isabel Leal
paper_authors._id: 65b0828cc11502dcf18b8702
paper_authors.name: Edward Lee
paper_authors._id: 65b0828cc11502dcf18b8703
paper_authors.name: Sergey Levine
paper_authors._id: 65b0828cc11502dcf18b8704
paper_authors.name: Yao Lu
paper_authors._id: 65b0828cc11502dcf18b8705
paper_authors.name: Isabel Leal
paper_authors._id: 65b0828cc11502dcf18b8706
paper_authors.name: Sharath Maddineni
paper_authors._id: 65b0828cc11502dcf18b8707
paper_authors.name: Kanishka Rao
paper_authors._id: 65b0828cc11502dcf18b8708
paper_authors.name: Dorsa Sadigh
paper_authors._id: 65b0828cc11502dcf18b8709
paper_authors.name: Pannag Sanketi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:49:17.408Z
paper_authors._id: 66b56f73fc66a466e6104bfb
paper_authors.avatarUrl: /avatars/74855aaf580144fafead4b93d6a24a46.svg
paper_authors.fullname: Pannag Sanketi
paper_authors.type: user
paper_authors.user: prsanky
paper_authors._id: 65b0828cc11502dcf18b870a
paper_authors.name: Pierre Sermanet
paper_authors._id: 65b0828cc11502dcf18b870b
paper_authors.name: Quan Vuong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:08:51.376Z
paper_authors._id: 63e1d6687fbb6ae4d4f4d980
paper_authors.avatarUrl: /avatars/4959319e69d69e03a31ff4be326e602a.svg
paper_authors.fullname: Quan Vuong
paper_authors.type: user
paper_authors.user: quintonx
paper_authors._id: 65b0828cc11502dcf18b870c
paper_authors.name: Stefan Welker
paper_authors._id: 65b0828cc11502dcf18b870d
paper_authors.name: Fei Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:11:23.433Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_authors._id: 65b0828cc11502dcf18b870e
paper_authors.name: Ted Xiao
paper_authors._id: 65b0828cc11502dcf18b870f
paper_authors.name: Peng Xu
paper_authors._id: 65b0828cc11502dcf18b8710
paper_authors.name: Steve Xu
paper_authors._id: 65b0828cc11502dcf18b8711
paper_authors.name: Zhuo Xu
paper_title: AutoRT: Embodied Foundation Models for Large Scale Orchestration of
    Robotic Agents
paper_summary: Foundation models that incorporate language, vision, and more recently
  actions have revolutionized the ability to harness internet scale data to
  reason about useful tasks. However, one of the key challenges of training
  embodied foundation models is the lack of data grounded in the physical world.
  In this paper, we propose AutoRT, a system that leverages existing foundation
  models to scale up the deployment of operational robots in completely unseen
  scenarios with minimal human supervision. AutoRT leverages vision-language
  models (VLMs) for scene understanding and grounding, and further uses large
  language models (LLMs) for proposing diverse and novel instructions to be
  performed by a fleet of robots. Guiding data collection by tapping into the
  knowledge of foundation models enables AutoRT to effectively reason about
  autonomy tradeoffs and safety while significantly scaling up data collection
  for robot learning. We demonstrate AutoRT proposing instructions to over 20
  robots across multiple buildings and collecting 77k real robot episodes via
  both teleoperation and autonomous robot policies. We experimentally show that
  such "in-the-wild" data collected by AutoRT is significantly more diverse, and
  that AutoRT's use of LLMs allows for instruction following data collection
  robots that can align to human preferences.
paper_ai_summary: AutoRT uses vision-language and large language models to enable operational robots to perform diverse tasks in unseen scenarios with minimal human supervision, enhancing data collection and alignment with human preferences.
paper_ai_keywords: vision-language models large language models embodied foundation models scene understanding guiding data collection autonomy tradeoffs safety instruction following data collection