title: GPT4Tools: Teaching Large Language Model to Use Tools via
    Self-instruction
summary: This paper aims to efficiently enable Large Language Models (LLMs) to use
  multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have
  shown great potential for tool usage through sophisticated prompt engineering.
  Nevertheless, these models typically rely on prohibitive computational costs
  and publicly inaccessible data. To address these challenges, we propose the
  GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and
  OPT, to use tools. It generates an instruction-following dataset by prompting
  an advanced teacher with various multi-modal contexts. By using the Low-Rank
  Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs
  to solve a range of visual problems, including visual comprehension and image
  generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to
  use tools, which is performed in both zero-shot and fine-tuning ways. Extensive
  experiments demonstrate the effectiveness of our method on various language
  models, which not only significantly improves the accuracy of invoking seen
  tools, but also enables the zero-shot capacity for unseen tools. The code and
  demo are available at https://github.com/StevenGrove/GPT4Tools.
paper_authors._id: 6476a1ce3d8d1cb79190b30a
paper_authors.name: Rui Yang
paper_authors._id: 6476a1ce3d8d1cb79190b30b
paper_authors.name: Lin Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-31T12:01:01.658Z
paper_authors._id: 6392a269e04c469baca3b154
paper_authors.avatarUrl: /avatars/6446beb55a06f5038e0055c4f4ee8c44.svg
paper_authors.fullname: linsong
paper_authors.type: user
paper_authors.user: stevengrove
paper_authors._id: 6476a1ce3d8d1cb79190b30c
paper_authors.name: Yanwei Li
paper_authors._id: 6476a1ce3d8d1cb79190b30d
paper_authors.name: Sijie Zhao
paper_authors._id: 6476a1ce3d8d1cb79190b30e
paper_authors.name: Yixiao Ge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-01T16:30:20.722Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 6476a1ce3d8d1cb79190b30f
paper_authors.name: Xiu Li
paper_authors._id: 6476a1ce3d8d1cb79190b310
paper_authors.name: Ying Shan
paper_title: GPT4Tools: Teaching Large Language Model to Use Tools via
    Self-instruction
paper_summary: This paper aims to efficiently enable Large Language Models (LLMs) to use
  multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have
  shown great potential for tool usage through sophisticated prompt engineering.
  Nevertheless, these models typically rely on prohibitive computational costs
  and publicly inaccessible data. To address these challenges, we propose the
  GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and
  OPT, to use tools. It generates an instruction-following dataset by prompting
  an advanced teacher with various multi-modal contexts. By using the Low-Rank
  Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs
  to solve a range of visual problems, including visual comprehension and image
  generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to
  use tools, which is performed in both zero-shot and fine-tuning ways. Extensive
  experiments demonstrate the effectiveness of our method on various language
  models, which not only significantly improves the accuracy of invoking seen
  tools, but also enables the zero-shot capacity for unseen tools. The code and
  demo are available at https://github.com/StevenGrove/GPT4Tools.
paper_ai_summary: GPT4Tools enables efficient tool usage by open-source LLMs through LoRA optimization, improving accuracy and zero-shot capacity for unseen tools.
paper_ai_keywords: Large Language Models LLMs ChatGPT GPT-4 self-instruct LLaMA OPT Low-Rank Adaptation LoRA visual comprehension image generation benchmark zero-shot fine-tuning