Retrieval Models Aren’t Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models

Abstract

Tool learning aims to augment large language models (LLMs) with diverse tools, enabling them to act as agents for solving practical tasks.Due to the limited context length of tool-using LLMs, adopting information retrieval (IR) models to select useful tools from large toolsets is a critical initial step.However, the performance of IR models in tool retrieval tasks remains underexplored and unclear.Most tooluse benchmarks simplify this step by manually pre-annotating a small set of relevant tools for each task, which is far from the real-world scenarios.In this paper, we propose TOOLRET, a heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks, and a corpus of 43k tools, collected from existing datasets.We benchmark six types of models on TOOL-RET.Surprisingly, even the models with strong performance in conventional IR benchmarks, exhibit poor performance on TOOLRET.This low retrieval quality degrades the task pass rate of tool-use LLMs.As a further step, we contribute a large-scale training dataset with over 200k instances, which substantially optimizes the tool retrieval ability of IR models. 1

Authors 7

1. Zhengliang Shi

   Shandong University

   Affiliation as printed

   Shandong University , Qingdao , China

2. Yuhan Wang

   Shandong University

   Affiliation as printed

   Shandong University , Qingdao , China

3. Lingyong Yan

   Baidu (China)

   Affiliation as printed

   Baidu Inc. , Beijing , China

4. Pengjie Ren

   Shandong University

   Affiliation as printed

   Shandong University , Qingdao , China

5. Shuaiqiang Wang

   Baidu (China)

   Affiliation as printed

   Baidu Inc. , Beijing , China

6. Dawei Yin

   Baidu (China)

   Affiliation as printed

   Baidu Inc. , Beijing , China

7. Zhaochun Ren Aachen

   Leiden University

   Affiliation as printed

   Leiden University , Leiden , The Netherlands