In this section, we describe the individual modalities that we
incorporate into PaLM-E, and how we set up their encoders.
We propose different architectural choices for each encoder
φ : O → X to map the corresponding modality into the
language embedding space. We investigate state estimation
vectors, Vision Transformers (ViTs) (Dosovitskiy et al.,
2020; Chen et al., 2022; Ryoo et al., 2021) for 2D image
features, and the 3D-aware Object Scene Representation
Transformer (OSRT) (Sajjadi et al., 2022a). In addition to
encoders that represent the input scene globally, we consider
object-centric representations that factor observations into
tokens that represent individual objects in the scene.
State estimation vectors. State vectors, e.g. from a robot
or a state estimate for objects, are perhaps the simplest to
input into PaLM-E. Let s ∈ RS be a vector describing the
state of the objects in a scene. For example, s could contain
the pose, size, color etc. of those objects. Then, the MLP
PaLM-E- An Embodied Multimodal Language Model