title: ToolTalk: Evaluating Tool-Usage in a Conversational Setting
summary: Large language models (LLMs) have displayed massive improvements in reason-
  ing and decision-making skills and can hold natural conversations with users.
  Many recent works seek to augment LLM-based assistants with external tools so
  they can access private or up-to-date information and carry out actions on
  behalf of users. To better measure the performance of these assistants, this
  paper introduces ToolTalk, a benchmark consisting of complex user intents re-
  quiring multi-step tool usage specified through dialogue. ToolTalk contains 28
  tools grouped into 7 plugins, and includes a complete simulated implementa-
  tion of each tool, allowing for fully automated evaluation of assistants that
  rely on execution feedback. ToolTalk also emphasizes tools that externally
  affect the world rather than only tools for referencing or searching
  information. We evaluate GPT-3.5 and GPT-4 on ToolTalk resulting in success
  rates of 26% and 50% respectively. Our analysis of the errors reveals three
  major categories and suggests some future directions for improvement. We
  release ToolTalk at https://github.com/microsoft/ToolTalk.
paper_authors._id: 655c157b1f8c93413b522a3b
paper_authors.name: Nicholas Farn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:54:51.694Z
paper_authors._id: 62f4193975277836ab9fd839
paper_authors.avatarUrl: /avatars/20ea02d36b17895cba9a21c9ef1120e7.svg
paper_authors.fullname: Nicholas Farn
paper_authors.type: user
paper_authors.user: nifarn
paper_authors._id: 655c157b1f8c93413b522a3c
paper_authors.name: Richard Shin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:54:58.253Z
paper_authors._id: 63cccaff6d080f9b7cfe9afb
paper_authors.avatarUrl: /avatars/c1124d1ba5fd116788f944962700ba16.svg
paper_authors.fullname: Richard Shin
paper_authors.type: user
paper_authors.user: rshin
paper_title: ToolTalk: Evaluating Tool-Usage in a Conversational Setting
paper_summary: Large language models (LLMs) have displayed massive improvements in reason-
  ing and decision-making skills and can hold natural conversations with users.
  Many recent works seek to augment LLM-based assistants with external tools so
  they can access private or up-to-date information and carry out actions on
  behalf of users. To better measure the performance of these assistants, this
  paper introduces ToolTalk, a benchmark consisting of complex user intents re-
  quiring multi-step tool usage specified through dialogue. ToolTalk contains 28
  tools grouped into 7 plugins, and includes a complete simulated implementa-
  tion of each tool, allowing for fully automated evaluation of assistants that
  rely on execution feedback. ToolTalk also emphasizes tools that externally
  affect the world rather than only tools for referencing or searching
  information. We evaluate GPT-3.5 and GPT-4 on ToolTalk resulting in success
  rates of 26% and 50% respectively. Our analysis of the errors reveals three
  major categories and suggests some future directions for improvement. We
  release ToolTalk at https://github.com/microsoft/ToolTalk.
paper_ai_summary: ToolTalk is a benchmark for evaluating LLM-based assistants using complex user intents requiring multi-step tool interactions, with success rates of 26% and 50% for GPT-3.5 and GPT-4 respectively.
paper_ai_keywords: Large language models LLMs ToolTalk benchmark user intents multi-step tool usage dialogue tools plugins simulated implementation execution feedback GPT-3.5 GPT-4