PaLM-E: An Embodied Multimodal Language Model

LLM pre-trained

(cid:51)(unfrozen)

SayCan (w/ oracle affordances)

φ

state
state
state

OSRT
state
state

state
state

state (w/o entity referrals)
ViT + TL (obj. centric)

ViT + TL (global)
ViT-4B (global)
ViT-4B generalist

state (w/o entity referrals)

state (w/o entity referrals)

state (8B LLM)
state (8B LLM)
state (62B LLM)

3 - 5
objects

6
objects

8
objects

6 objects +
OOD tasks

(cid:51)
(cid:55)

(cid:51)
(cid:51)
(cid:51)
(cid:51)
(cid:51)
(cid:51)
(cid:51)

(cid:55)
(cid:51)
(cid:51)

(cid:55)
(cid:51)
(cid:51)

(cid:55)
(cid:51)
(cid:51)

q1
-

100.0
100.0
100.0
100.0
99.6

99.6
20.4
100.0
77.7
18.4
100.0
60.0

-
-
-

-
-
-

q2
-

99.3
98.8
98.4
98.8
98.7
60.7
98.2
97.1
99.1
39.2
98.5
83.7
27.1
98.3
67.1

0

49.3
48.7

q3
-

98.5
100.0
99.7
97.5
98.4
90.8
99.4
100.0
100.0
71.4
94.0
93.6
38.1
95.3
94.1

0

89.8
92.5

q4
-

99.8
97.6
98.5
98.1
96.8
94.3
99.0
98.9
98.8
85.2
89.3
91.0
87.5
89.8
81.2
72.0
68.5
88.1
PaLM-E- An Embodied Multimodal Language Model