Figure 2: PaLM-E-562B can do zero-shot multimodal chain-of-thought reasoning, can tell visually-conditioned jokes given an image, and
demonstrates an array of robot-relevant multimodal-informed capabilities including perception, visually-grounded dialogue, and planning.
PaLM-E also generalizes, zero-shot, to multi-image prompts despite only being trained on single-image prompts. PaLM-E can also
perform math given an image with textually-interleaved handwritten numbers. In addition, the model can perform, zero-shot, question and
answering on temporally-annotated egocentric vision, similar to what was shown in (Zeng et al., 2022) but end-to-end all in one model.
PaLM-E- An Embodied Multimodal Language Model