1. Introduction
Large language models (LLMs) demonstrate strong reason-
ing capabilities across various domains, including dialogue
(Glaese et al., 2022; Thoppilan et al., 2022), step-by-step
reasoning (Wei et al., 2022; Kojima et al., 2022), math prob-
lem solving (Lewkowycz et al., 2022; Polu et al., 2022), and
code writing (Chen et al., 2021a). However, a limitation of
such models for inference in the real world is the issue of
grounding: while training LLMs on massive textual data
may lead to representations that relate to our physical world,
connecting those representations to real-world visual and
physical sensor modalities is essential to solving a wider
range of grounded real-world problems in computer vision
and robotics (Tellex et al., 2020). Previous work (Ahn et al.,
2022) interfaces the output of LLMs with learned robotic
policies and affordance functions to make decisions, but is
limited in that the LLM itself is only provided with textual
PaLM-E- An Embodied Multimodal Language Model