tecture (Locatello et al., 2020). Based on SRT (Sajjadi
et al., 2022b), OSRT learns 3D-centric neural scene rep-
resentations on in-domain data through a novel view syn-
thesis task. Its scene representations consist of object slots
oj = ¯φOSRT(I1:v)j ∈ R¯k. We project each of these slots
1:m = ψ( ¯φOSRT(I1:v)j) with an MLP ψ. Note that
into xj
individual objects are always tokenized into multiple em-
beddings each, i.e. ψ : R¯k → Rm×k for OSRT maps into
m-many embeddings.
Entity referrals. For embodied planning tasks, PaLM-E
must be able to reference objects in its generated plan. In
many cases, including the majority of our experiments,
objects in a scene can be identiﬁed in natural language
by some of their unique properties. However,
there
also exist settings where objects are not easily identiﬁ-
able by language in few words, e.g. if there are multi-
ple blocks on a table of the same color at different loca-
tions. For object-centric representations such as OSRT, we
PaLM-E- An Embodied Multimodal Language Model