Replication and Exploration of Generative Retrieval over Dynamic Corpora

International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 3325–3334

Abstract

Generative retrieval (GR) has emerged as a promising paradigm in information retrieval (IR). However, most existing GR models are developed and evaluated using a static document collection, and their performance in dynamic corpora where document collections evolve continuously is rarely studied. In this paper, we first reproduce and systematically evaluate various representative GR approaches over dynamic corpora. Through extensive experiments, we reveal that existing GR models with text-based docids show superior generalization to unseen documents. We observe that the more fine-grained the docid design in the GR model, the better its performance over dynamic corpora, surpassing BM25 and even being comparable to dense retrieval methods. While GR models with numeric-based docids show high efficiency, their performance drops significantly over dynamic corpora. Furthermore, our experiments find that the underperformance of numeric-based docids is partly due to their excessive tendency toward the initial document set, which likely results from overfitting on the training set. We then conduct an in-depth analysis of the best-performing GR methods. We identify three critical advantages of text-based docids in dynamic corpora: (i) Semantic alignment with language models' pretrained knowledge (ii) Fine-grained docid design, and (iii) High lexical diversity. Building on these insights, we finally propose a novel multi-docid design that leverages both the efficiency of numeric-based docids and the effectiveness of text-based docids, achieving improved performance in dynamic corpus without requiring additional retraining. Our work offers empirical evidence for advancing GR methods over dynamic corpora and paves the way for developing more generalized yet efficient GR models in real-world search engines.

Authors 9

1. Zhen Zhang

   Shandong University

   Affiliation as printed

   Shandong University, Qingdao, China

2. Xinyu Ma

   Baidu (China)

   Affiliation as printed

   Baidu Inc., Beijing, China

3. Weiwei Sun

   Carnegie Mellon University

   Affiliation as printed

   Carnegie Mellon University, Pittsburgh, USA

4. Pengjie Ren

   Shandong University

   Affiliation as printed

   Shandong University, Qingdao, China

5. Zhumin Chen

   Shandong University

   Affiliation as printed

   Shandong University, Qingdao, China

6. Shuaiqiang Wang

   Baidu (China)

   Affiliation as printed

   Baidu Inc., Beijing, China

7. Dawei Yin

   Baidu (China)

   Affiliation as printed

   Baidu Inc., Beijing, China

8. Maarten de Rijke

   University of Amsterdam

   Affiliation as printed

   University of Amsterdam, Amsterdam, Netherlands

9. Zhaochun Ren Aachen

   Leiden University

   Affiliation as printed

   Leiden University, Leiden, Netherlands