title: Foundation Models in Robotics: Applications, Challenges, and the Future
summary: We survey applications of pretrained foundation models in robotics.
  Traditional deep learning models in robotics are trained on small datasets
  tailored for specific tasks, which limits their adaptability across diverse
  applications. In contrast, foundation models pretrained on internet-scale data
  appear to have superior generalization capabilities, and in some instances
  display an emergent ability to find zero-shot solutions to problems that are
  not present in the training data. Foundation models may hold the potential to
  enhance various components of the robot autonomy stack, from perception to
  decision-making and control. For example, large language models can generate
  code or provide common sense reasoning, while vision-language models enable
  open-vocabulary visual recognition. However, significant open research
  challenges remain, particularly around the scarcity of robot-relevant training
  data, safety guarantees and uncertainty quantification, and real-time
  execution. In this survey, we study recent papers that have used or built
  foundation models to solve robotics problems. We explore how foundation models
  contribute to improving robot capabilities in the domains of perception,
  decision-making, and control. We discuss the challenges hindering the adoption
  of foundation models in robot autonomy and provide opportunities and potential
  pathways for future advancements. The GitHub project corresponding to this
  paper (Preliminary release. We are committed to further enhancing and updating
  this work to ensure its quality and relevance) can be found here:
  https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models
paper_authors._id: 657a651ec222807b36e8641d
paper_authors.name: Roya Firoozi
paper_authors._id: 657a651ec222807b36e8641e
paper_authors.name: Johnathan Tucker
paper_authors._id: 657a651ec222807b36e8641f
paper_authors.name: Stephen Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:37:58.119Z
paper_authors._id: 64bf3b4a36eb058cd931a22e
paper_authors.avatarUrl: /avatars/0946015f01b8774029dd694f558eab2c.svg
paper_authors.fullname: Stephen Tian
paper_authors.type: user
paper_authors.user: s-tian
paper_authors._id: 657a651ec222807b36e86420
paper_authors.name: Anirudha Majumdar
paper_authors._id: 657a651ec222807b36e86421
paper_authors.name: Jiankai Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:38:10.641Z
paper_authors._id: 63b5d1bb0d5913eee4869c0d
paper_authors.avatarUrl: /avatars/ae397f54bbb3debc1f7903f4c6959ae6.svg
paper_authors.fullname: Jiankai Sun
paper_authors.type: user
paper_authors.user: zhenv5
paper_authors._id: 657a651ec222807b36e86422
paper_authors.name: Weiyu Liu
paper_authors._id: 657a651ec222807b36e86423
paper_authors.name: Yuke Zhu
paper_authors._id: 657a651ec222807b36e86424
paper_authors.name: Shuran Song
paper_authors._id: 657a651ec222807b36e86425
paper_authors.name: Ashish Kapoor
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:39:00.107Z
paper_authors._id: 62a2eadd3621790e22a49b47
paper_authors.avatarUrl: /avatars/b055657d60fffe310c05428ece2646e3.svg
paper_authors.fullname: Ashish Kapoor
paper_authors.type: user
paper_authors.user: ashishkapoor
paper_authors._id: 657a651ec222807b36e86426
paper_authors.name: Karol Hausman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:39:08.507Z
paper_authors._id: 64c93e623a3f428da67d9310
paper_authors.avatarUrl: /avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
paper_authors.fullname: Karol Hausman
paper_authors.type: user
paper_authors.user: KarolHausman
paper_authors._id: 657a651ec222807b36e86427
paper_authors.name: Brian Ichter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:39:15.386Z
paper_authors._id: 633a26ab474cfeb1a864dc56
paper_authors.avatarUrl: /avatars/cd40accd894fc78810fc0d5108f413e9.svg
paper_authors.fullname: Brian I
paper_authors.type: user
paper_authors.user: brianichter
paper_authors._id: 657a651ec222807b36e86428
paper_authors.name: Danny Driess
paper_authors._id: 657a651ec222807b36e86429
paper_authors.name: Jiajun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:39:39.168Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_authors._id: 657a651ec222807b36e8642a
paper_authors.name: Cewu Lu
paper_authors._id: 657a651ec222807b36e8642b
paper_authors.name: Mac Schwager
paper_title: Foundation Models in Robotics: Applications, Challenges, and the Future
paper_summary: We survey applications of pretrained foundation models in robotics.
  Traditional deep learning models in robotics are trained on small datasets
  tailored for specific tasks, which limits their adaptability across diverse
  applications. In contrast, foundation models pretrained on internet-scale data
  appear to have superior generalization capabilities, and in some instances
  display an emergent ability to find zero-shot solutions to problems that are
  not present in the training data. Foundation models may hold the potential to
  enhance various components of the robot autonomy stack, from perception to
  decision-making and control. For example, large language models can generate
  code or provide common sense reasoning, while vision-language models enable
  open-vocabulary visual recognition. However, significant open research
  challenges remain, particularly around the scarcity of robot-relevant training
  data, safety guarantees and uncertainty quantification, and real-time
  execution. In this survey, we study recent papers that have used or built
  foundation models to solve robotics problems. We explore how foundation models
  contribute to improving robot capabilities in the domains of perception,
  decision-making, and control. We discuss the challenges hindering the adoption
  of foundation models in robot autonomy and provide opportunities and potential
  pathways for future advancements. The GitHub project corresponding to this
  paper (Preliminary release. We are committed to further enhancing and updating
  this work to ensure its quality and relevance) can be found here:
  https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models
paper_ai_summary: Pretrained foundation models, trained on internet-scale data, offer improved generalization and can enhance various aspects of robot autonomy, including perception, decision-making, and control, despite challenges in robotics-specific data, safety, and real-time execution.
paper_ai_keywords: pretrained foundation models internet-scale data generalization capabilities zero-shot solutions robot autonomy stack large language models common sense reasoning vision-language models open-vocabulary visual recognition robot-relevant training data safety guarantees uncertainty quantification real-time execution