title: Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for
    Large Language Models
summary: Tool learning aims to augment large language models (LLMs) with diverse
  tools, enabling them to act as agents for solving practical tasks. Due to the
  limited context length of tool-using LLMs, adopting information retrieval (IR)
  models to select useful tools from large toolsets is a critical initial step.
  However, the performance of IR models in tool retrieval tasks remains
  underexplored and unclear. Most tool-use benchmarks simplify this step by
  manually pre-annotating a small set of relevant tools for each task, which is
  far from the real-world scenarios. In this paper, we propose ToolRet, a
  heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks,
  and a corpus of 43k tools, collected from existing datasets. We benchmark six
  types of models on ToolRet. Surprisingly, even the models with strong
  performance in conventional IR benchmarks, exhibit poor performance on ToolRet.
  This low retrieval quality degrades the task pass rate of tool-use LLMs. As a
  further step, we contribute a large-scale training dataset with over 200k
  instances, which substantially optimizes the tool retrieval ability of IR
  models.
paper_authors._id: 67c92e9c746bbcdbdfa8ebd4
paper_authors.name: Zhengliang Shi
paper_authors._id: 67c92e9c746bbcdbdfa8ebd5
paper_authors.name: Yuhan Wang
paper_authors._id: 67c92e9c746bbcdbdfa8ebd6
paper_authors.name: Lingyong Yan
paper_authors._id: 67c92e9c746bbcdbdfa8ebd7
paper_authors.name: Pengjie Ren
paper_authors._id: 67c92e9c746bbcdbdfa8ebd8
paper_authors.name: Shuaiqiang Wang
paper_authors._id: 67c92e9c746bbcdbdfa8ebd9
paper_authors.name: Dawei Yin
paper_authors._id: 67c92e9c746bbcdbdfa8ebda
paper_authors.name: Zhaochun Ren
paper_title: Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for
    Large Language Models
paper_summary: Tool learning aims to augment large language models (LLMs) with diverse
  tools, enabling them to act as agents for solving practical tasks. Due to the
  limited context length of tool-using LLMs, adopting information retrieval (IR)
  models to select useful tools from large toolsets is a critical initial step.
  However, the performance of IR models in tool retrieval tasks remains
  underexplored and unclear. Most tool-use benchmarks simplify this step by
  manually pre-annotating a small set of relevant tools for each task, which is
  far from the real-world scenarios. In this paper, we propose ToolRet, a
  heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks,
  and a corpus of 43k tools, collected from existing datasets. We benchmark six
  types of models on ToolRet. Surprisingly, even the models with strong
  performance in conventional IR benchmarks, exhibit poor performance on ToolRet.
  This low retrieval quality degrades the task pass rate of tool-use LLMs. As a
  further step, we contribute a large-scale training dataset with over 200k
  instances, which substantially optimizes the tool retrieval ability of IR
  models.
paper_ai_keywords: heterogeneous tool retrieval benchmark retrieval tasks tool-use benchmarks strong performance in conventional IR benchmarks