vectors with the same dimension as the embedding space of
the language tokens. The continuous information is hence
injected into the language model in an analogous way to
language tokens. PaLM-E is a decoder-only LLM that gen-
erates textual completions autoregressively given a preﬁx or
prompt. We call our model PaLM-E, since we use PaLM
(Chowdhery et al., 2022) as the pre-trained language model,
and make it Embodied.
The inputs to PaLM-E consist of text and (multiple) con-
tinuous observations. The multimodal tokens correspond-
ing to these observations are interleaved with the text
to form multi-modal sentences. An example of such a
multi-modal sentence is Q: What happened between
<img 1> and <img 2>? where <img i> represents an em-
bedding of an image. The output of PaLM-E is text gen-
erated auto-regressively by the model, which could be an
answer to a question, or a sequence of decisions produced by
PaLM-E in textual form that should be executed by a robot.
PaLM-E- An Embodied Multimodal Language Model