The

emergence

of multimodal

LLM
agents (Wang et al., 2023; Furuta et al., 2023;
Brohan et al., 2022, 2023; Reed et al., 2022),
capable of processing various inputs including text,
images, audio, and video, has further broadened
the scope of LLM applications. This versatility
is particularly beneficial for LLM-based agents,
enabling them to interact more effectively with
their environment and complete more complex
tasks, be it completing household tasks in a
physical world (Ahn et al., 2022), generating 3D
assets via procedural tool use (Sun et al., 2023), or
mastering over 600 tasks across different domains
at the same time (Reed et al., 2022). Our research
contributes to this area by focusing on an agent
designed to operate smartphone applications. This
agent’s ability to interpret screenshots from the
operating system demonstrates its flexibility and
adaptability, making it a valuable tool in a wide

3
AppAgents