MAIR: A Massive Benchmark for Evaluating Instructed Retrieval

arXiv (Cornell University)

Abstract

Recent information retrieval (IR) models are pre-trained and instruction-tuned on massive datasets and tasks, enabling them to perform well on a wide range of tasks and potentially generalize to unseen tasks with instructions. However, existing IR benchmarks focus on a limited scope of tasks, making them insufficient for evaluating the latest IR models. In this paper, we propose MAIR (Massive Instructed Retrieval Benchmark), a heterogeneous IR benchmark that includes 126 distinct IR tasks across 6 domains, collected from existing datasets. We benchmark state-of-the-art instruction-tuned text embedding models and re-ranking models. Our experiments reveal that instruction-tuned models generally achieve superior performance compared to non-instruction-tuned models on MAIR. Additionally, our results suggest that current instruction-tuned text embedding models and re-ranking models still lack effectiveness in specific long-tail tasks. MAIR is publicly available at https://github.com/sunnweiwei/Mair.

Authors 8

1. Weiwei Sun

   Carnegie Mellon University

   Affiliation as printed

   Carnegie Mellon University

2. Zhengliang Shi

   Shandong University

   Affiliation as printed

   Shandong University

3. Kesheng John Wu

   Soochow University

   Affiliation as printed

   Soochow University

4. Lingyong Yan

   Baidu (China)

   Affiliation as printed

   Baidu Inc

5. Xinyu Ma

   Baidu (China)

   Affiliation as printed

   Baidu Inc

6. Yiding Liu

   Baidu (China)

   Affiliation as printed

   Baidu Inc

7. Dawei Yin

   Baidu (China)

   Affiliation as printed

   Baidu Inc

8. Zhaochun Ren Aachen

   Leiden University

   Affiliation as printed

   Leiden University