input encoders, and if so, how different-modality encoders
compare. In this case, the encoder has to produce embed-
ding vectors such that the frozen LLM is grounded on the
observations, and also propagate information to the LLM
about the capabilities of an embodiment. Training such en-
codings can be understood as a form of input-conditioned
soft-prompting (Tsimpoukelli et al., 2021), in relation to nor-
mal soft prompts (Lester et al., 2021). In experiments with
φOSRT, we also freeze the slot representation, i.e. we only
update the small projector ψ which serves as the interface
between OSRT and the LLM.
Co-training across tasks. In our experiments, we investi-
gate the effects of co-training our models on a variety of
diverse data. The “full mixture”, see App. A, consists pri-
marily of a diverse set of internet-scale vision-and-language
data, from a variety of tasks. The sampling frequencies are
set such that only 8.9% of the full mixture is embodied data,
PaLM-E- An Embodied Multimodal Language Model