set such that only 8.9% of the full mixture is embodied data,
and there are several tasks for each embodiment.
6. Experiments
Our experiments consider diverse robotic (mobile) manip-
ulation tasks across three different robot embodiments, in
simulation and with two different real robots. We refer to
https://palm-e.github.io for videos showing the
capabilities of PaLM-E on those tasks. Although not the
focus of our work, we evaluate PaLM-E also on general
vision-language tasks such as visual-question-answering
(VQA), image captioning, and established language model-
ing tasks.
We split our experimental investigation into two broad cate-
gories. First, we compare the different input representations
from Sec. 4 with respect to performance, generalization, and
data-efﬁciency. The second thread of experiments focuses
on one architecture, the main PaLM-E version, consisting
of a pre-trained ViT and PaLM language model that takes
in raw images as the continuous inputs. Here we show
PaLM-E- An Embodied Multimodal Language Model