title: ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world
    APIs
summary: Despite the advancements of open-source large language models (LLMs) and
  their variants, e.g., LLaMA and Vicuna, they remain significantly limited in
  performing higher-level tasks, such as following human instructions to use
  external tools (APIs). This is because current instruction tuning largely
  focuses on basic language tasks instead of the tool-use domain. This is in
  contrast to state-of-the-art (SOTA) LLMs, e.g., ChatGPT, which have
  demonstrated excellent tool-use capabilities but are unfortunately closed
  source. To facilitate tool-use capabilities within open-source LLMs, we
  introduce ToolLLM, a general tool-use framework of data construction, model
  training and evaluation. We first present ToolBench, an instruction-tuning
  dataset for tool use, which is created automatically using ChatGPT.
  Specifically, we collect 16,464 real-world RESTful APIs spanning 49 categories
  from RapidAPI Hub, then prompt ChatGPT to generate diverse human instructions
  involving these APIs, covering both single-tool and multi-tool scenarios.
  Finally, we use ChatGPT to search for a valid solution path (chain of API
  calls) for each instruction. To make the searching process more efficient, we
  develop a novel depth-first search-based decision tree (DFSDT), enabling LLMs
  to evaluate multiple reasoning traces and expand the search space. We show that
  DFSDT significantly enhances the planning and reasoning capabilities of LLMs.
  For efficient tool-use assessment, we develop an automatic evaluator: ToolEval.
  We fine-tune LLaMA on ToolBench and obtain ToolLLaMA. Our ToolEval reveals that
  ToolLLaMA demonstrates a remarkable ability to execute complex instructions and
  generalize to unseen APIs, and exhibits comparable performance to ChatGPT. To
  make the pipeline more practical, we devise a neural API retriever to recommend
  appropriate APIs for each instruction, negating the need for manual API
  selection.
paper_authors._id: 64c85d053d5a0dfed5b65fc9
paper_authors.name: Yujia Qin
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-08-01T01:18:13.843Z
paper_authors._id: 643f37cce9d063936912048b
paper_authors.avatarUrl: /avatars/25822ea5676a79b2e1ddf08d5fc2226c.svg
paper_authors.fullname: Yujia Qin
paper_authors.type: user
paper_authors.user: YujiaHi
paper_authors._id: 64c85d053d5a0dfed5b65fca
paper_authors.name: Shihao Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T07:44:43.257Z
paper_authors._id: 6241d6b4bdf59d13c0a8dd86
paper_authors.avatarUrl: /avatars/f729f0bdcfbd971babfa760d8de8403c.svg
paper_authors.fullname: Shihao Liang
paper_authors.type: user
paper_authors.user: pooruss
paper_authors._id: 64c85d053d5a0dfed5b65fcb
paper_authors.name: Yining Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-02T07:39:26.901Z
paper_authors._id: 636a0803fbc31ee68fed16bd
paper_authors.avatarUrl: /avatars/331073d0fee901b2b9d93287fabe64fc.svg
paper_authors.fullname: yining ye
paper_authors.type: user
paper_authors.user: yeyn2001
paper_authors._id: 64c85d053d5a0dfed5b65fcc
paper_authors.name: Kunlun Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T07:46:35.841Z
paper_authors._id: 64c090a9f613170e7be93d2f
paper_authors.avatarUrl: /avatars/ccbdf444e1f2386d2281e8e42059ebb0.svg
paper_authors.fullname: KunlunZhu
paper_authors.type: user
paper_authors.user: KunlunZhu
paper_authors._id: 64c85d053d5a0dfed5b65fcd
paper_authors.name: Lan Yan
paper_authors._id: 64c85d053d5a0dfed5b65fce
paper_authors.name: Yaxi Lu
paper_authors._id: 64c85d053d5a0dfed5b65fcf
paper_authors.name: Yankai Lin
paper_authors._id: 64c85d053d5a0dfed5b65fd0
paper_authors.name: Xin Cong
paper_authors._id: 64c85d053d5a0dfed5b65fd1
paper_authors.name: Xiangru Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T11:17:54.591Z
paper_authors._id: 63357c608adfa81faf2ac180
paper_authors.avatarUrl: /avatars/ae0314c644f882251baf59b9134fd36f.svg
paper_authors.fullname: Xiangru Tang
paper_authors.type: user
paper_authors.user: RTT1
paper_authors._id: 64c85d053d5a0dfed5b65fd2
paper_authors.name: Bill Qian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T07:48:25.080Z
paper_authors._id: 641fa2e0863b87326f4a7fd5
paper_authors.avatarUrl: /avatars/7b29c20f942d83c08f63f04a85c15c13.svg
paper_authors.fullname: Bill Qian
paper_authors.type: user
paper_authors.user: lilbillbiscuit
paper_authors._id: 64c85d053d5a0dfed5b65fd3
paper_authors.name: Sihan Zhao
paper_authors._id: 64c85d053d5a0dfed5b65fd4
paper_authors.name: Runchu Tian
paper_authors._id: 64c85d053d5a0dfed5b65fd5
paper_authors.name: Ruobing Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T09:02:01.538Z
paper_authors._id: 6622443b9b0614a760dd8123
paper_authors.avatarUrl: /avatars/acb6c1c9c429af1112530dcf76a8e420.svg
paper_authors.fullname: Ruobing Xie
paper_authors.type: user
paper_authors.user: Ruobing-Xie
paper_authors._id: 64c85d053d5a0dfed5b65fd6
paper_authors.name: Jie Zhou
paper_authors._id: 64c85d053d5a0dfed5b65fd7
paper_authors.name: Mark Gerstein
paper_authors._id: 64c85d053d5a0dfed5b65fd8
paper_authors.name: Dahai Li
paper_authors._id: 64c85d053d5a0dfed5b65fd9
paper_authors.name: Zhiyuan Liu
paper_authors._id: 64c85d053d5a0dfed5b65fda
paper_authors.name: Maosong Sun
paper_title: ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world
    APIs
paper_summary: Despite the advancements of open-source large language models (LLMs) and
  their variants, e.g., LLaMA and Vicuna, they remain significantly limited in
  performing higher-level tasks, such as following human instructions to use
  external tools (APIs). This is because current instruction tuning largely
  focuses on basic language tasks instead of the tool-use domain. This is in
  contrast to state-of-the-art (SOTA) LLMs, e.g., ChatGPT, which have
  demonstrated excellent tool-use capabilities but are unfortunately closed
  source. To facilitate tool-use capabilities within open-source LLMs, we
  introduce ToolLLM, a general tool-use framework of data construction, model
  training and evaluation. We first present ToolBench, an instruction-tuning
  dataset for tool use, which is created automatically using ChatGPT.
  Specifically, we collect 16,464 real-world RESTful APIs spanning 49 categories
  from RapidAPI Hub, then prompt ChatGPT to generate diverse human instructions
  involving these APIs, covering both single-tool and multi-tool scenarios.
  Finally, we use ChatGPT to search for a valid solution path (chain of API
  calls) for each instruction. To make the searching process more efficient, we
  develop a novel depth-first search-based decision tree (DFSDT), enabling LLMs
  to evaluate multiple reasoning traces and expand the search space. We show that
  DFSDT significantly enhances the planning and reasoning capabilities of LLMs.
  For efficient tool-use assessment, we develop an automatic evaluator: ToolEval.
  We fine-tune LLaMA on ToolBench and obtain ToolLLaMA. Our ToolEval reveals that
  ToolLLaMA demonstrates a remarkable ability to execute complex instructions and
  generalize to unseen APIs, and exhibits comparable performance to ChatGPT. To
  make the pipeline more practical, we devise a neural API retriever to recommend
  appropriate APIs for each instruction, negating the need for manual API
  selection.
paper_ai_keywords: ToolLLM ToolBench instruction-tuning RESTful APIs depth-first search-based decision tree (DFSDT) ToolEval neural API retriever