title: AppWorld: A Controllable World of Apps and People for Benchmarking
    Interactive Coding Agents
summary: Autonomous agents that address day-to-day digital tasks (e.g., ordering
  groceries for a household), must not only operate multiple apps (e.g., notes,
  messaging, shopping app) via APIs, but also generate rich code with complex
  control flow in an iterative manner based on their interaction with the
  environment. However, existing benchmarks for tool use are inadequate, as they
  only cover tasks that require a simple sequence of API calls.
    To remedy this gap, we built AppWorld Engine, a high-quality
  execution environment (60K lines of code) of 9 day-to-day apps operable via 457
  APIs and populated with realistic digital activities simulating the lives of
  ~100 fictitious users. We then created AppWorld Benchmark (40K lines
  of code), a suite of 750 natural, diverse, and challenging autonomous agent
  tasks requiring rich and interactive code generation. It supports robust
  programmatic evaluation with state-based unit tests, allowing for different
  ways of completing a task while also checking for unexpected changes, i.e.,
  collateral damage. The state-of-the-art LLM, GPT-4o, solves only ~49% of our
  'normal' tasks and ~30% of 'challenge' tasks, while other models solve at least
  16% fewer. This highlights the benchmark's difficulty and AppWorld's potential
  to push the frontiers of interactive coding agents. The project website is
  available at https://appworld.dev/.
paper_authors._id: 66a6fe95e2d12966a3f644f3
paper_authors.name: Harsh Trivedi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T07:24:45.106Z
paper_authors._id: 5f1e2cbfcb8f993fa01f47a4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670052569285-5f1e2cbfcb8f993fa01f47a4.jpeg
paper_authors.fullname: Harsh
paper_authors.type: user
paper_authors.user: harshtrivedi
paper_authors._id: 66a6fe95e2d12966a3f644f4
paper_authors.name: Tushar Khot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T07:24:52.186Z
paper_authors._id: 62cdf2dbaac2c91c95581830
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cdf2dbaac2c91c95581830/N3NuO7j53qkauJOf3kkRS.jpeg
paper_authors.fullname: Tushar Khot
paper_authors.type: user
paper_authors.user: tusharkhot
paper_authors._id: 66a6fe95e2d12966a3f644f5
paper_authors.name: Mareike Hartmann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T07:24:57.932Z
paper_authors._id: 5e8341e130dc073f817a2bd4
paper_authors.avatarUrl: /avatars/3de44416dcdc67fc3da3150da33ca9b3.svg
paper_authors.fullname: Mareike Hartmann
paper_authors.type: user
paper_authors.user: mahartmann
paper_authors._id: 66a6fe95e2d12966a3f644f6
paper_authors.name: Ruskin Manku
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T07:25:03.355Z
paper_authors._id: 66958c29d4ca2767b9c41005
paper_authors.avatarUrl: /avatars/c81b65c1ad345c48c252773ea78b7607.svg
paper_authors.fullname: Ruskin Raj Manku
paper_authors.type: user
paper_authors.user: ruskinmanku
paper_authors._id: 66a6fe95e2d12966a3f644f7
paper_authors.name: Vinty Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T07:25:18.010Z
paper_authors._id: 6459827c5e44a749d5a4f3a7
paper_authors.avatarUrl: /avatars/5bccd0f95739e4984861c678c4ad7c9c.svg
paper_authors.fullname: Vinty Dong
paper_authors.type: user
paper_authors.user: vdong
paper_authors._id: 66a6fe95e2d12966a3f644f8
paper_authors.name: Edward Li
paper_authors._id: 66a6fe95e2d12966a3f644f9
paper_authors.name: Shashank Gupta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-29T07:03:31.292Z
paper_authors._id: 5e9d15b74957053f60648a34
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e9d15b74957053f60648a34/_Nt3HWtz6moVQIfa5ieIJ.jpeg
paper_authors.fullname: Shashank Gupta
paper_authors.type: user
paper_authors.user: shatu
paper_authors._id: 66a6fe95e2d12966a3f644fa
paper_authors.name: Ashish Sabharwal
paper_authors._id: 66a6fe95e2d12966a3f644fb
paper_authors.name: Niranjan Balasubramanian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T07:25:50.165Z
paper_authors._id: 655eb80660f1a93e901bc926
paper_authors.avatarUrl: /avatars/35305994ac1c28eb9395810dbc705561.svg
paper_authors.fullname: Niranjan Balasubramanian
paper_authors.type: user
paper_authors.user: niranjan-b
paper_title: AppWorld: A Controllable World of Apps and People for Benchmarking
    Interactive Coding Agents
paper_summary: Autonomous agents that address day-to-day digital tasks (e.g., ordering
  groceries for a household), must not only operate multiple apps (e.g., notes,
  messaging, shopping app) via APIs, but also generate rich code with complex
  control flow in an iterative manner based on their interaction with the
  environment. However, existing benchmarks for tool use are inadequate, as they
  only cover tasks that require a simple sequence of API calls.
    To remedy this gap, we built AppWorld Engine, a high-quality
  execution environment (60K lines of code) of 9 day-to-day apps operable via 457
  APIs and populated with realistic digital activities simulating the lives of
  ~100 fictitious users. We then created AppWorld Benchmark (40K lines
  of code), a suite of 750 natural, diverse, and challenging autonomous agent
  tasks requiring rich and interactive code generation. It supports robust
  programmatic evaluation with state-based unit tests, allowing for different
  ways of completing a task while also checking for unexpected changes, i.e.,
  collateral damage. The state-of-the-art LLM, GPT-4o, solves only ~49% of our
  'normal' tasks and ~30% of 'challenge' tasks, while other models solve at least
  16% fewer. This highlights the benchmark's difficulty and AppWorld's potential
  to push the frontiers of interactive coding agents. The project website is
  available at https://appworld.dev/.
paper_ai_summary: A comprehensive benchmark and execution environment for evaluating autonomous agents performing complex, realistic digital tasks across multiple apps through rich code generation.
paper_ai_keywords: AppWorld Engine AppWorld Benchmark state-based unit tests interactive code generation autonomous agents LLM GPT-4o