title: MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language
    Models
summary: Large Language Models (LLMs) have displayed massive improvements in reasoning
  and decision-making skills and can hold natural conversations with users.
  Recently, many tool-use benchmark datasets have been proposed. However,
  existing datasets have the following limitations: (1). Insufficient evaluation
  scenarios (e.g., only cover limited tool-use scenes). (2). Extensive evaluation
  costs (e.g., GPT API costs). To address these limitations, in this work, we
  propose a multi-granularity tool-use benchmark for large language models called
  MTU-Bench. For the "multi-granularity" property, our MTU-Bench covers five tool
  usage scenes (i.e., single-turn and single-tool, single-turn and multiple-tool,
  multiple-turn and single-tool, multiple-turn and multiple-tool, and
  out-of-distribution tasks). Besides, all evaluation metrics of our MTU-Bench
  are based on the prediction results and the ground truth without using any GPT
  or human evaluation metrics. Moreover, our MTU-Bench is collected by
  transforming existing high-quality datasets to simulate real-world tool usage
  scenarios, and we also propose an instruction dataset called MTU-Instruct data
  to enhance the tool-use abilities of existing LLMs. Comprehensive experimental
  results demonstrate the effectiveness of our MTU-Bench. Code and data will be
  released at https: //github.com/MTU-Bench-Team/MTU-Bench.git.
paper_authors._id: 670f270f3240681ca35b0f9e
paper_authors.name: Pei Wang
paper_authors._id: 670f270f3240681ca35b0f9f
paper_authors.name: Yanan Wu
paper_authors._id: 670f270f3240681ca35b0fa0
paper_authors.name: Zekun Wang
paper_authors._id: 670f270f3240681ca35b0fa1
paper_authors.name: Jiaheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:03:31.306Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 670f270f3240681ca35b0fa2
paper_authors.name: Xiaoshuai Song
paper_authors._id: 670f270f3240681ca35b0fa3
paper_authors.name: Zhongyuan Peng
paper_authors._id: 670f270f3240681ca35b0fa4
paper_authors.name: Ken Deng
paper_authors._id: 670f270f3240681ca35b0fa5
paper_authors.name: Chenchen Zhang
paper_authors._id: 670f270f3240681ca35b0fa6
paper_authors.name: Jiakai Wang
paper_authors._id: 670f270f3240681ca35b0fa7
paper_authors.name: Junran Peng
paper_authors._id: 670f270f3240681ca35b0fa8
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:55:40.883Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 670f270f3240681ca35b0fa9
paper_authors.name: Hangyu Guo
paper_authors._id: 670f270f3240681ca35b0faa
paper_authors.name: Zhaoxiang Zhang
paper_authors._id: 670f270f3240681ca35b0fab
paper_authors.name: Wenbo Su
paper_authors._id: 670f270f3240681ca35b0fac
paper_authors.name: Bo Zheng
paper_title: MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language
    Models
paper_summary: Large Language Models (LLMs) have displayed massive improvements in reasoning
  and decision-making skills and can hold natural conversations with users.
  Recently, many tool-use benchmark datasets have been proposed. However,
  existing datasets have the following limitations: (1). Insufficient evaluation
  scenarios (e.g., only cover limited tool-use scenes). (2). Extensive evaluation
  costs (e.g., GPT API costs). To address these limitations, in this work, we
  propose a multi-granularity tool-use benchmark for large language models called
  MTU-Bench. For the "multi-granularity" property, our MTU-Bench covers five tool
  usage scenes (i.e., single-turn and single-tool, single-turn and multiple-tool,
  multiple-turn and single-tool, multiple-turn and multiple-tool, and
  out-of-distribution tasks). Besides, all evaluation metrics of our MTU-Bench
  are based on the prediction results and the ground truth without using any GPT
  or human evaluation metrics. Moreover, our MTU-Bench is collected by
  transforming existing high-quality datasets to simulate real-world tool usage
  scenarios, and we also propose an instruction dataset called MTU-Instruct data
  to enhance the tool-use abilities of existing LLMs. Comprehensive experimental
  results demonstrate the effectiveness of our MTU-Bench. Code and data will be
  released at https: //github.com/MTU-Bench-Team/MTU-Bench.git.
paper_ai_keywords: Large Language Models (LLMs) dataset improvements reasoning decision-making skills natural conversations tool-use benchmark datasets evaluation scenarios GPT API costs multi-granularity tool-use benchmark MTU-Bench single-turn and single-tool single-turn and multiple-tool multiple-turn and single-tool multiple-turn and multiple-tool out-of-distribution tasks prediction results ground truth real-world tool usage scenarios MTU-Instruct data experimental results effectiveness