Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for
  Large Language Models
Tool learning aims to augment large language models (LLMs) with diverse
tools, enabling them to act as agents for solving practical tasks. Due to the
limited context length of tool-using LLMs, adopting information retrieval (IR)
models to select useful tools from large toolsets is a critical initial step.
However, the performance of IR models in tool retrieval tasks remains
underexplored and unclear. Most tool-use benchmarks simplify this step by
manually pre-annotating a small set of relevant tools for each task, which is
far from the real-world scenarios. In this paper, we propose ToolRet, a
heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks,
and a corpus of 43k tools, collected from existing datasets. We benchmark six
types of models on ToolRet. Surprisingly, even the models with strong
performance in conventional IR benchmarks, exhibit poor performance on ToolRet.
This low retrieval quality degrades the task pass rate of tool-use LLMs. As a
further step, we contribute a large-scale training dataset with over 200k
instances, which substantially optimizes the tool retrieval ability of IR
models.
67c92e9c746bbcdbdfa8ebd4
Zhengliang Shi
67c92e9c746bbcdbdfa8ebd5
Yuhan Wang
67c92e9c746bbcdbdfa8ebd6
Lingyong Yan
67c92e9c746bbcdbdfa8ebd7
Pengjie Ren
67c92e9c746bbcdbdfa8ebd8
Shuaiqiang Wang
67c92e9c746bbcdbdfa8ebd9
Dawei Yin
67c92e9c746bbcdbdfa8ebda
Zhaochun Ren
Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for
  Large Language Models
Tool learning aims to augment large language models (LLMs) with diverse
tools, enabling them to act as agents for solving practical tasks. Due to the
limited context length of tool-using LLMs, adopting information retrieval (IR)
models to select useful tools from large toolsets is a critical initial step.
However, the performance of IR models in tool retrieval tasks remains
underexplored and unclear. Most tool-use benchmarks simplify this step by
manually pre-annotating a small set of relevant tools for each task, which is
far from the real-world scenarios. In this paper, we propose ToolRet, a
heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks,
and a corpus of 43k tools, collected from existing datasets. We benchmark six
types of models on ToolRet. Surprisingly, even the models with strong
performance in conventional IR benchmarks, exhibit poor performance on ToolRet.
This low retrieval quality degrades the task pass rate of tool-use LLMs. As a
further step, we contribute a large-scale training dataset with over 200k
instances, which substantially optimizes the tool retrieval ability of IR
models.
A heterogeneous tool retrieval benchmark (ToolRet) is introduced to evaluate the performance of information retrieval models in selecting tools for large language models, highlighting their suboptimal performance and providing a large-scale training dataset to improve retrieval quality.
tool learning large language models information retrieval tool retrieval benchmark heterogeneous benchmark retrieval tasks toolset pre-annotating task pass rate