LANCER: LLM Reranking for Nugget Coverage

arXiv (Cornell University)

Abstract

Unlike short-form retrieval-augmented generation (RAG), such as factoid question answering, long-form RAG requires retrieval to provide documents covering a wide range of relevant information. Automated report generation exemplifies this setting: it requires not only relevant information but also a more elaborate response with comprehensive information. Yet, existing retrieval methods are primarily optimized for relevance ranking rather than information coverage. To address this limitation, we propose LANCER, an LLM-based reranking method for nugget coverage. LANCER predicts what sub-questions should be answered to satisfy an information need, predicts which documents answer these sub-questions, and reranks documents in order to provide a ranked list covering as many information nuggets as possible. Our empirical results show that LANCER enhances the quality of retrieval as measured by nugget coverage metrics, achieving higher $α$-nDCG and information coverage than other LLM-based reranking methods. Our oracle analysis further reveals that sub-question generation plays an essential role.

Authors 5

1. Jia-Huei Ju

   University of Amsterdam

   Affiliation as printed

   University of Amsterdam , The Netherlands

2. François G. Landry

   Université de Moncton

   Affiliation as printed

   Université de Moncton , Canada

3. Eugene Yang

   Johns Hopkins University

   Affiliation as printed

   Johns Hopkins University , USA

4. Suzan Verberne Aachen

   Leiden University

   Affiliation as printed

   Leiden University , The Netherlands

5. Andrew Yates

   Johns Hopkins University

   Affiliation as printed

   Johns Hopkins University , USA