title: GTA: A Benchmark for General Tool Agents
summary: Significant focus has been placed on integrating large language models (LLMs)
  with various tools in developing general-purpose agents. This poses a challenge
  to LLMs' tool-use capabilities. However, there are evident gaps between
  existing tool-use evaluations and real-world scenarios. Current evaluations
  often use AI-generated queries, single-step tasks, dummy tools, and text-only
  interactions, failing to reveal the agents' real-world problem-solving
  abilities effectively. To address this, we propose GTA, a benchmark for General
  Tool Agents, featuring three main aspects: (i) Real user queries: human-written
  queries with simple real-world objectives but implicit tool-use, requiring the
  LLM to reason the suitable tools and plan the solution steps. (ii) Real
  deployed tools: an evaluation platform equipped with tools across perception,
  operation, logic, and creativity categories to evaluate the agents' actual task
  execution performance. (iii) Real multimodal inputs: authentic image files,
  such as spatial scenes, web page screenshots, tables, code snippets, and
  printed/handwritten materials, used as the query contexts to align with
  real-world scenarios closely. We design 229 real-world tasks and executable
  tool chains to evaluate mainstream LLMs. Our findings show that real-world user
  queries are challenging for existing LLMs, with GPT-4 completing less than 50%
  of the tasks and most LLMs achieving below 25%. This evaluation reveals the
  bottlenecks in the tool-use capabilities of current LLMs in real-world
  scenarios, which provides future direction for advancing general-purpose tool
  agents. The code and dataset are available at
  https://github.com/open-compass/GTA.
paper_authors._id: 66909d0698c0444e3a7e408a
paper_authors.name: Jize Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-13T07:39:49.961Z
paper_authors._id: 66323843a6ef9d3afdf11fbb
paper_authors.avatarUrl: /avatars/2fe9313634fa30dfa3ca36de5cd094f0.svg
paper_authors.fullname: Jize
paper_authors.type: user
paper_authors.user: Jize1
paper_authors._id: 66909d0698c0444e3a7e408b
paper_authors.name: Zerun Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T08:43:33.636Z
paper_authors._id: 634fb993685e43e02d2544a2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634fb993685e43e02d2544a2/L9MuXYlXMBuQgnwXd4lCU.jpeg
paper_authors.fullname: mzr1996
paper_authors.type: user
paper_authors.user: mzr1996
paper_authors._id: 66909d0698c0444e3a7e408c
paper_authors.name: Yining Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T09:58:52.289Z
paper_authors._id: 649988726677f66c2b486392
paper_authors.avatarUrl: /avatars/b649a77370660e129726e29504daba34.svg
paper_authors.fullname: Yining Li
paper_authors.type: user
paper_authors.user: ly015
paper_authors._id: 66909d0698c0444e3a7e408d
paper_authors.name: Songyang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T09:57:07.530Z
paper_authors._id: 630716d11801ecc7d2595021
paper_authors.avatarUrl: /avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
paper_authors.fullname: Songyang Zhang
paper_authors.type: user
paper_authors.user: zsytony
paper_authors._id: 66909d0698c0444e3a7e408e
paper_authors.name: Cailian Chen
paper_authors._id: 66909d0698c0444e3a7e408f
paper_authors.name: Kai Chen
paper_authors._id: 66909d0698c0444e3a7e4090
paper_authors.name: Xinyi Le
paper_title: GTA: A Benchmark for General Tool Agents
paper_summary: Significant focus has been placed on integrating large language models (LLMs)
  with various tools in developing general-purpose agents. This poses a challenge
  to LLMs' tool-use capabilities. However, there are evident gaps between
  existing tool-use evaluations and real-world scenarios. Current evaluations
  often use AI-generated queries, single-step tasks, dummy tools, and text-only
  interactions, failing to reveal the agents' real-world problem-solving
  abilities effectively. To address this, we propose GTA, a benchmark for General
  Tool Agents, featuring three main aspects: (i) Real user queries: human-written
  queries with simple real-world objectives but implicit tool-use, requiring the
  LLM to reason the suitable tools and plan the solution steps. (ii) Real
  deployed tools: an evaluation platform equipped with tools across perception,
  operation, logic, and creativity categories to evaluate the agents' actual task
  execution performance. (iii) Real multimodal inputs: authentic image files,
  such as spatial scenes, web page screenshots, tables, code snippets, and
  printed/handwritten materials, used as the query contexts to align with
  real-world scenarios closely. We design 229 real-world tasks and executable
  tool chains to evaluate mainstream LLMs. Our findings show that real-world user
  queries are challenging for existing LLMs, with GPT-4 completing less than 50%
  of the tasks and most LLMs achieving below 25%. This evaluation reveals the
  bottlenecks in the tool-use capabilities of current LLMs in real-world
  scenarios, which provides future direction for advancing general-purpose tool
  agents. The code and dataset are available at
  https://github.com/open-compass/GTA.
paper_ai_keywords: large language models (LLMs) general-purpose agents tool-use capabilities tool-use evaluations AI-generated queries single-step tasks dummy tools real user queries implicit tool-use suitable tools solution steps real deployed tools evaluation platform perception operation logic creativity real multimodal inputs spatial scenes web page screenshots tables code snippets printed/handwritten materials query contexts real-world tasks executable tool chains mainstream LLMs GPT-4