Reactivity. Reactivity in an agent refers to its ability to respond rapidly to immediate changes and
stimuli in its environment [9]. This implies that the agent can perceive alterations in its surroundings
and promptly take appropriate actions. Traditionally, the perceptual space of language models
has been confined to textual inputs, while the action space has been limited to textual outputs.
However, researchers have demonstrated the potential to expand the perceptual space of LLMs using
multimodal fusion techniques, enabling them to rapidly process visual and auditory information from
the environment [25; 118; 119]. Similarly, it’s also feasible to expand the action space of LLMs
through embodiment techniques [120; 121] and tool usage [92; 94]. These advancements enable
LLMs to effectively interact with the real-world physical environment and carry out tasks within it.
One major challenge is that LLM-based agents, when performing non-textual actions, require an
TheRiseandPotentialofLargeLanguageModel BasedAgents