title: Tool Documentation Enables Zero-Shot Tool-Usage with Large Language
    Models
summary: Today, large language models (LLMs) are taught to use new tools by providing
  a few demonstrations of the tool's usage. Unfortunately, demonstrations are
  hard to acquire, and can result in undesirable biased usage if the wrong
  demonstration is chosen. Even in the rare scenario that demonstrations are
  readily available, there is no principled selection protocol to determine how
  many and which ones to provide. As tasks grow more complex, the selection
  search grows combinatorially and invariably becomes intractable. Our work
  provides an alternative to demonstrations: tool documentation. We advocate the
  use of tool documentation, descriptions for the individual tool usage, over
  demonstrations. We substantiate our claim through three main empirical findings
  on 6 tasks across both vision and language modalities. First, on existing
  benchmarks, zero-shot prompts with only tool documentation are sufficient for
  eliciting proper tool usage, achieving performance on par with few-shot
  prompts. Second, on a newly collected realistic tool-use dataset with hundreds
  of available tool APIs, we show that tool documentation is significantly more
  valuable than demonstrations, with zero-shot documentation significantly
  outperforming few-shot without documentation. Third, we highlight the benefits
  of tool documentations by tackling image generation and video tracking using
  just-released unseen state-of-the-art models as tools. Finally, we highlight
  the possibility of using tool documentation to automatically enable new
  applications: by using nothing more than the documentation of GroundingDino,
  Stable Diffusion, XMem, and SAM, LLMs can re-invent the functionalities of the
  just-released Grounded-SAM and Track Anything models.
paper_authors._id: 64c9c3f2f7f4ccb5ea66fc39
paper_authors.name: Cheng-Yu Hsieh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T20:05:16.142Z
paper_authors._id: 6408e8663461c51cf73480e4
paper_authors.avatarUrl: /avatars/7e4cbdb4b933181c673c9dc9ad041767.svg
paper_authors.fullname: Cheng-Yu Hsieh
paper_authors.type: user
paper_authors.user: cydhsieh01
paper_authors._id: 64c9c3f2f7f4ccb5ea66fc3a
paper_authors.name: Si-An Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T08:17:44.711Z
paper_authors._id: 64cafdf84dcdaead7af64425
paper_authors.avatarUrl: /avatars/6888d0eefa84ba218f2151922179f2f5.svg
paper_authors.fullname: Si-An Chen
paper_authors.type: user
paper_authors.user: sianchen
paper_authors._id: 64c9c3f2f7f4ccb5ea66fc3b
paper_authors.name: Chun-Liang Li
paper_authors._id: 64c9c3f2f7f4ccb5ea66fc3c
paper_authors.name: Yasuhisa Fujii
paper_authors._id: 64c9c3f2f7f4ccb5ea66fc3d
paper_authors.name: Alexander Ratner
paper_authors._id: 64c9c3f2f7f4ccb5ea66fc3e
paper_authors.name: Chen-Yu Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-08T13:45:17.962Z
paper_authors._id: 64d1fe784dfd5df7076bfd1a
paper_authors.avatarUrl: /avatars/e2bb108d1f6b1383f0e3f263c4747b3d.svg
paper_authors.fullname: Chen-Yu Lee
paper_authors.type: user
paper_authors.user: chenyulee
paper_authors._id: 64c9c3f2f7f4ccb5ea66fc3f
paper_authors.name: Ranjay Krishna
paper_authors._id: 64c9c3f2f7f4ccb5ea66fc40
paper_authors.name: Tomas Pfister
paper_title: Tool Documentation Enables Zero-Shot Tool-Usage with Large Language
    Models
paper_summary: Today, large language models (LLMs) are taught to use new tools by providing
  a few demonstrations of the tool's usage. Unfortunately, demonstrations are
  hard to acquire, and can result in undesirable biased usage if the wrong
  demonstration is chosen. Even in the rare scenario that demonstrations are
  readily available, there is no principled selection protocol to determine how
  many and which ones to provide. As tasks grow more complex, the selection
  search grows combinatorially and invariably becomes intractable. Our work
  provides an alternative to demonstrations: tool documentation. We advocate the
  use of tool documentation, descriptions for the individual tool usage, over
  demonstrations. We substantiate our claim through three main empirical findings
  on 6 tasks across both vision and language modalities. First, on existing
  benchmarks, zero-shot prompts with only tool documentation are sufficient for
  eliciting proper tool usage, achieving performance on par with few-shot
  prompts. Second, on a newly collected realistic tool-use dataset with hundreds
  of available tool APIs, we show that tool documentation is significantly more
  valuable than demonstrations, with zero-shot documentation significantly
  outperforming few-shot without documentation. Third, we highlight the benefits
  of tool documentations by tackling image generation and video tracking using
  just-released unseen state-of-the-art models as tools. Finally, we highlight
  the possibility of using tool documentation to automatically enable new
  applications: by using nothing more than the documentation of GroundingDino,
  Stable Diffusion, XMem, and SAM, LLMs can re-invent the functionalities of the
  just-released Grounded-SAM and Track Anything models.
paper_ai_keywords: large language models (LLMs) tool documentation demonstration bounding box task documentation zero-shot prompts few-shot prompts vision language modalities realistic characteristic-use dataset API image generation video tracking state-of-the-art models GroundingDino Stable Diffusion XMem SAM Grounded-SAM Track Anything models