Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection

Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP), pp. 139–149

Abstract

Search methods based on Pretrained Language Models (PLM) have demonstrated great effectiveness gains compared to statistical and early neural ranking models. However, fine-tuning PLM-based rankers requires a great amount of annotated training data. Annotating data involves a large manual effort and thus is expensive, especially in domain specific tasks. In this paper we investigate fine-tuning PLM-based rankers under limited training data and budget. We investigate two scenarios: fine-tuning a ranker from scratch, and domain adaptation starting with a ranker already fine-tuned on general data, and continuing fine-tuning on a target dataset.

Authors 5

1. Sophia Althammer

   TU Wien

   Affiliation as printed

   Information System Engineering, TU Wien, Austria

2. Guido Zuccon

   The University of Queensland

   Affiliation as printed

   ITEE, The University of Queensland, Australia

3. Sebastian Hofstätter
   Affiliation as printed

   Cohere, Austria

4. Suzan Verberne Aachen

   Leiden University

   Affiliation as printed

   LIACS, Leiden University, Netherlands

5. Allan Hanbury

   TU Wien

   Affiliation as printed

   TU Wien, Austria