AppWorld: A Controllable World of Apps and People for Benchmarking
  Interactive Coding Agents
Autonomous agents that address day-to-day digital tasks (e.g., ordering
groceries for a household), must not only operate multiple apps (e.g., notes,
messaging, shopping app) via APIs, but also generate rich code with complex
control flow in an iterative manner based on their interaction with the
environment. However, existing benchmarks for tool use are inadequate, as they
only cover tasks that require a simple sequence of API calls.
  To remedy this gap, we built AppWorld Engine, a high-quality
execution environment (60K lines of code) of 9 day-to-day apps operable via 457
APIs and populated with realistic digital activities simulating the lives of
~100 fictitious users. We then created AppWorld Benchmark (40K lines
of code), a suite of 750 natural, diverse, and challenging autonomous agent
tasks requiring rich and interactive code generation. It supports robust
programmatic evaluation with state-based unit tests, allowing for different
ways of completing a task while also checking for unexpected changes, i.e.,
collateral damage. The state-of-the-art LLM, GPT-4o, solves only ~49% of our
'normal' tasks and ~30% of 'challenge' tasks, while other models solve at least
16% fewer. This highlights the benchmark's difficulty and AppWorld's potential
to push the frontiers of interactive coding agents. The project website is
available at https://appworld.dev/.
66a6fe95e2d12966a3f644f3
Harsh Trivedi
admin_assigned
2024-07-29T07:24:45.106Z
5f1e2cbfcb8f993fa01f47a4
https://cdn-avatars.huggingface.co/v1/production/uploads/1670052569285-5f1e2cbfcb8f993fa01f47a4.jpeg
Harsh
user
harshtrivedi
66a6fe95e2d12966a3f644f4
Tushar Khot
admin_assigned
2024-07-29T07:24:52.186Z
62cdf2dbaac2c91c95581830
https://cdn-avatars.huggingface.co/v1/production/uploads/62cdf2dbaac2c91c95581830/N3NuO7j53qkauJOf3kkRS.jpeg
Tushar Khot
user
tusharkhot
66a6fe95e2d12966a3f644f5
Mareike Hartmann
admin_assigned
2024-07-29T07:24:57.932Z
5e8341e130dc073f817a2bd4
/avatars/3de44416dcdc67fc3da3150da33ca9b3.svg
Mareike Hartmann
user
mahartmann
66a6fe95e2d12966a3f644f6
Ruskin Manku
admin_assigned
2024-07-29T07:25:03.355Z
66958c29d4ca2767b9c41005
/avatars/c81b65c1ad345c48c252773ea78b7607.svg
Ruskin Raj Manku
user
ruskinmanku
66a6fe95e2d12966a3f644f7
Vinty Dong
admin_assigned
2024-07-29T07:25:18.010Z
6459827c5e44a749d5a4f3a7
/avatars/5bccd0f95739e4984861c678c4ad7c9c.svg
Vinty Dong
user
vdong
66a6fe95e2d12966a3f644f8
Edward Li
66a6fe95e2d12966a3f644f9
Shashank Gupta
claimed_verified
2024-07-29T07:03:31.292Z
5e9d15b74957053f60648a34
https://cdn-avatars.huggingface.co/v1/production/uploads/5e9d15b74957053f60648a34/_Nt3HWtz6moVQIfa5ieIJ.jpeg
Shashank Gupta
user
shatu
66a6fe95e2d12966a3f644fa
Ashish Sabharwal
claimed_verified
2025-07-22T07:57:43.302Z
687e87f9b193d0f4c8d605c0
/avatars/1e2fa294cbb2017e380fae6695e35db1.svg
Ashish Sabharwal
user
ashish333
66a6fe95e2d12966a3f644fb
Niranjan Balasubramanian
admin_assigned
2024-07-29T07:25:50.165Z
655eb80660f1a93e901bc926
/avatars/35305994ac1c28eb9395810dbc705561.svg
Niranjan Balasubramanian
user
niranjan-b
AppWorld: A Controllable World of Apps and People for Benchmarking
  Interactive Coding Agents
Autonomous agents that address day-to-day digital tasks (e.g., ordering
groceries for a household), must not only operate multiple apps (e.g., notes,
messaging, shopping app) via APIs, but also generate rich code with complex
control flow in an iterative manner based on their interaction with the
environment. However, existing benchmarks for tool use are inadequate, as they
only cover tasks that require a simple sequence of API calls.
  To remedy this gap, we built AppWorld Engine, a high-quality
execution environment (60K lines of code) of 9 day-to-day apps operable via 457
APIs and populated with realistic digital activities simulating the lives of
~100 fictitious users. We then created AppWorld Benchmark (40K lines
of code), a suite of 750 natural, diverse, and challenging autonomous agent
tasks requiring rich and interactive code generation. It supports robust
programmatic evaluation with state-based unit tests, allowing for different
ways of completing a task while also checking for unexpected changes, i.e.,
collateral damage. The state-of-the-art LLM, GPT-4o, solves only ~49% of our
'normal' tasks and ~30% of 'challenge' tasks, while other models solve at least
16% fewer. This highlights the benchmark's difficulty and AppWorld's potential
to push the frontiers of interactive coding agents. The project website is
available at https://appworld.dev/.
A comprehensive benchmark and execution environment for evaluating autonomous agents performing complex, realistic digital tasks across multiple apps through rich code generation.
AppWorld Engine AppWorld Benchmark state-based unit tests interactive code generation autonomous agents LLM GPT-4o