title: ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models
    in Multi-Hop Tool Use
summary: Effective evaluation of multi-hop tool use is critical for analyzing the
  understanding, reasoning, and function-calling capabilities of large language
  models (LLMs). However, progress has been hindered by a lack of reliable
  evaluation datasets. To address this, we present ToolHop, a dataset comprising
  995 user queries and 3,912 associated tools, specifically designed for rigorous
  evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful
  interdependencies, locally executable tools, detailed feedback, and verifiable
  answers through a novel query-driven data construction approach that includes
  tool creation, document refinement, and code generation. We evaluate 14 LLMs
  across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and
  GPT), uncovering significant challenges in handling multi-hop tool-use
  scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%,
  underscoring substantial room for improvement. Further analysis reveals
  variations in tool-use strategies for various families, offering actionable
  insights to guide the development of more effective approaches. Code and data
  can be found in https://huggingface.co/bytedance-research/ToolHop.
paper_authors._id: 677cab009909f23ac4dc1d0e
paper_authors.name: Junjie Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T13:02:00.881Z
paper_authors._id: 66384be673c2c55f2ded89fa
paper_authors.avatarUrl: /avatars/1d8721074f0f51fab405f81474f2035f.svg
paper_authors.fullname: Junjie Ye
paper_authors.type: user
paper_authors.user: Junjie-Ye
paper_authors._id: 677cab009909f23ac4dc1d0f
paper_authors.name: Zhengyin Du
paper_authors._id: 677cab009909f23ac4dc1d10
paper_authors.name: Xuesong Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:29:49.648Z
paper_authors._id: 656d8ec3fa91c81609822b96
paper_authors.avatarUrl: /avatars/56f6eed2cdd4706db45ea6b20059265d.svg
paper_authors.fullname: Yao Xuesong
paper_authors.type: user
paper_authors.user: NathanYao
paper_authors._id: 677cab009909f23ac4dc1d11
paper_authors.name: Weijian Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:42:22.956Z
paper_authors._id: 6511f4fdc915724a65380f78
paper_authors.avatarUrl: /avatars/b7cf645e0a035e3a3683ee1fd01380ed.svg
paper_authors.fullname: Weijian Lin
paper_authors.type: user
paper_authors.user: maverick1994
paper_authors._id: 677cab009909f23ac4dc1d12
paper_authors.name: Yufei Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:52:15.015Z
paper_authors._id: 642957104e073875f6a5ddd0
paper_authors.avatarUrl: /avatars/32b706d35c5ff52932c5029b94caa7b9.svg
paper_authors.fullname: Yufei Xu
paper_authors.type: user
paper_authors.user: yfxu
paper_authors._id: 677cab009909f23ac4dc1d13
paper_authors.name: Zehui Chen
paper_authors._id: 677cab009909f23ac4dc1d14
paper_authors.name: Zaiyuan Wang
paper_authors._id: 677cab009909f23ac4dc1d15
paper_authors.name: Sining Zhu
paper_authors._id: 677cab009909f23ac4dc1d16
paper_authors.name: Zhiheng Xi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T13:26:27.253Z
paper_authors._id: 653a6e5cae155b92bae77b74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
paper_authors.fullname: Zhiheng Xi
paper_authors.type: user
paper_authors.user: WooooDyy
paper_authors._id: 677cab009909f23ac4dc1d17
paper_authors.name: Siyu Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T13:26:42.842Z
paper_authors._id: 62d62b333bf5e059f7d2b286
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668513815771-62d62b333bf5e059f7d2b286.jpeg
paper_authors.fullname: Siyu Yuan
paper_authors.type: user
paper_authors.user: siyuyuan
paper_authors._id: 677cab009909f23ac4dc1d18
paper_authors.name: Tao Gui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T13:26:50.647Z
paper_authors._id: 65b71b9e8bb894c96cc7707b
paper_authors.avatarUrl: /avatars/78e2a833054e89bcc34ddc6e789677d5.svg
paper_authors.fullname: Tao Gui
paper_authors.type: user
paper_authors.user: guixiaotao
paper_authors._id: 677cab009909f23ac4dc1d19
paper_authors.name: Qi Zhang
paper_authors._id: 677cab009909f23ac4dc1d1a
paper_authors.name: Xuanjing Huang
paper_authors._id: 677cab009909f23ac4dc1d1b
paper_authors.name: Jiechao Chen
paper_title: ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models
    in Multi-Hop Tool Use
paper_summary: Effective evaluation of multi-hop tool use is critical for analyzing the
  understanding, reasoning, and function-calling capabilities of large language
  models (LLMs). However, progress has been hindered by a lack of reliable
  evaluation datasets. To address this, we present ToolHop, a dataset comprising
  995 user queries and 3,912 associated tools, specifically designed for rigorous
  evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful
  interdependencies, locally executable tools, detailed feedback, and verifiable
  answers through a novel query-driven data construction approach that includes
  tool creation, document refinement, and code generation. We evaluate 14 LLMs
  across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and
  GPT), uncovering significant challenges in handling multi-hop tool-use
  scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%,
  underscoring substantial room for improvement. Further analysis reveals
  variations in tool-use strategies for various families, offering actionable
  insights to guide the development of more effective approaches. Code and data
  can be found in https://huggingface.co/bytedance-research/ToolHop.
paper_ai_summary: ToolHop, a new dataset for evaluating multi-hop tool use in large language models, highlights significant challenges and variations in tool-use strategies across different model families.
paper_ai_keywords: multi-hop tool use large language models (LLMs) ToolHop user queries tool creation document refinement code generation accuracy GPT-4o model families LLaMA3.1 Qwen2.5 Gemini1.5 Claude3.5 GPT