title: ToolSandbox: A Stateful, Conversational, Interactive Evaluation
    Benchmark for LLM Tool Use Capabilities
summary: Recent large language models (LLMs) advancements sparked a growing research
  interest in tool assisted LLMs solving real-world challenges, which calls for
  comprehensive evaluation of tool-use capabilities. While previous works focused
  on either evaluating over stateless web services (RESTful API), based on a
  single turn user prompt, or an off-policy dialog trajectory, ToolSandbox
  includes stateful tool execution, implicit state dependencies between tools, a
  built-in user simulator supporting on-policy conversational evaluation and a
  dynamic evaluation strategy for intermediate and final milestones over an
  arbitrary trajectory. We show that open source and proprietary models have a
  significant performance gap, and complex tasks like State Dependency,
  Canonicalization and Insufficient Information defined in ToolSandbox are
  challenging even the most capable SOTA LLMs, providing brand-new insights into
  tool-use LLM capabilities. ToolSandbox evaluation framework is released at
  https://github.com/apple/ToolSandbox
paper_authors._id: 66b97d1874d4fb4d570ee3c7
paper_authors.name: Jiarui Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T11:16:26.743Z
paper_authors._id: 64bad07938953777fe3ed5e9
paper_authors.avatarUrl: /avatars/91c07a176d8d51111873751e82a4e0ea.svg
paper_authors.fullname: Jiarui Lu
paper_authors.type: user
paper_authors.user: jiarui-lu
paper_authors._id: 66b97d1874d4fb4d570ee3c8
paper_authors.name: Thomas Holleis
paper_authors._id: 66b97d1874d4fb4d570ee3c9
paper_authors.name: Yizhe Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T11:16:52.951Z
paper_authors._id: 64d1284c59503263d91b4707
paper_authors.avatarUrl: /avatars/12d3799af150b538ad87874fdbf49dd1.svg
paper_authors.fullname: Yizhe Zhang
paper_authors.type: user
paper_authors.user: yizheapple
paper_authors._id: 66b97d1874d4fb4d570ee3ca
paper_authors.name: Bernhard Aumayer
paper_authors._id: 66b97d1874d4fb4d570ee3cb
paper_authors.name: Feng Nan
paper_authors._id: 66b97d1874d4fb4d570ee3cc
paper_authors.name: Felix Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T11:17:20.075Z
paper_authors._id: 6468582ba9b4610868a654ab
paper_authors.avatarUrl: /avatars/1f15b922a45e43e24905f9df98346ca5.svg
paper_authors.fullname: Felix Bai
paper_authors.type: user
paper_authors.user: fbai
paper_authors._id: 66b97d1874d4fb4d570ee3cd
paper_authors.name: Shuang Ma
paper_authors._id: 66b97d1874d4fb4d570ee3ce
paper_authors.name: Shen Ma
paper_authors._id: 66b97d1874d4fb4d570ee3cf
paper_authors.name: Mengyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T11:17:44.954Z
paper_authors._id: 6530b8bdcb5ad115c394e7cd
paper_authors.avatarUrl: /avatars/5621f2394e6c382c09147185b56d4154.svg
paper_authors.fullname: Mengyu Li
paper_authors.type: user
paper_authors.user: molee1004
paper_authors._id: 66b97d1874d4fb4d570ee3d0
paper_authors.name: Guoli Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T11:17:50.580Z
paper_authors._id: 605b7f42935268bc086131ba
paper_authors.avatarUrl: /avatars/a55109f714b33f9d59d69011ddeb0b9f.svg
paper_authors.fullname: Guoli Yin
paper_authors.type: user
paper_authors.user: gyin94
paper_authors._id: 66b97d1874d4fb4d570ee3d1
paper_authors.name: Zirui Wang
paper_authors._id: 66b97d1874d4fb4d570ee3d2
paper_authors.name: Ruoming Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T11:18:19.204Z
paper_authors._id: 654ef8ad3fe6c0b1f871942f
paper_authors.avatarUrl: /avatars/8d3689b9bf57c7c8060ac510a1839158.svg
paper_authors.fullname: Ruoming Pang
paper_authors.type: user
paper_authors.user: ruoming
paper_title: ToolSandbox: A Stateful, Conversational, Interactive Evaluation
    Benchmark for LLM Tool Use Capabilities
paper_summary: Recent large language models (LLMs) advancements sparked a growing research
  interest in tool assisted LLMs solving real-world challenges, which calls for
  comprehensive evaluation of tool-use capabilities. While previous works focused
  on either evaluating over stateless web services (RESTful API), based on a
  single turn user prompt, or an off-policy dialog trajectory, ToolSandbox
  includes stateful tool execution, implicit state dependencies between tools, a
  built-in user simulator supporting on-policy conversational evaluation and a
  dynamic evaluation strategy for intermediate and final milestones over an
  arbitrary trajectory. We show that open source and proprietary models have a
  significant performance gap, and complex tasks like State Dependency,
  Canonicalization and Insufficient Information defined in ToolSandbox are
  challenging even the most capable SOTA LLMs, providing brand-new insights into
  tool-use LLM capabilities. ToolSandbox evaluation framework is released at
  https://github.com/apple/ToolSandbox
paper_ai_keywords: stateless web services RESTful API stateful tool execution implicit state Dependencies on-policy conversational evaluation dynamic evaluation strategy intermediate and final milestones arbitrary trajectory performance gap State Dependency Canonicalization Insufficient Information SOTA LLMs tool-use LLM capabilities