Embodied actions for LLM-based agents. Depending on the agents’ level of autonomy in a task
or the complexity of actions, there are several fundamental LLM-based embodied actions, primarily
including observation, manipulation, and navigation.

• Observation. Observation constitutes the primary ways by which the agent acquires environmental
information and updates states, playing a crucial role in enhancing the efficiency of subsequent
embodied actions. As mentioned in §3.2, observation by embodied agents primarily occurs in
environments with various inputs, which are ultimately converged into a multimodal signal. A
common approach entails a pre-trained Vision Transformer (ViT) used as the alignment module for
text and visual information and special tokens are marked to denote the positions of multimodal
data [120; 332; 121]. Soundspaces [377] proposes the identification of physical spatial geometric

22
TheRiseandPotentialofLargeLanguageModel BasedAgents