Measuring Bias in a Ranked List using Term-based Representations

arXiv (Cornell University)

Abstract

In most recent studies, gender bias in document ranking is evaluated with the NFaiRR metric, which measures bias in a ranked list based on an aggregation over the unbiasedness scores of each ranked document. This perspective in measuring the bias of a ranked list has a key limitation: individual documents of a ranked list might be biased while the ranked list as a whole balances the groups' representations. To address this issue, we propose a novel metric called TExFAIR (term exposure-based fairness), which is based on two new extensions to a generic fairness evaluation framework, attention-weighted ranking fairness (AWRF). TExFAIR assesses fairness based on the term-based representation of groups in a ranked list: (i) an explicit definition of associating documents to groups based on probabilistic term-level associations, and (ii) a rank-biased discounting factor (RBDF) for counting non-representative documents towards the measurement of the fairness of a ranked list. We assess TExFAIR on the task of measuring gender bias in passage ranking, and study the relationship between TExFAIR and NFaiRR. Our experiments show that there is no strong correlation between TExFAIR and NFaiRR, which indicates that TExFAIR measures a different dimension of fairness than NFaiRR. With TExFAIR, we extend the AWRF framework to allow for the evaluation of fairness in settings with term-based representations of groups in documents in a ranked list.

Authors 5

1. Amin Abolghasemi Aachen

   Leiden University

   Affiliation as printed

   Leiden University , The Netherlands

2. Leif A. Azzopardi

   University of Strathclyde

   Affiliation as printed

   University of Strathclyde , UK

3. Arian Askari Aachen

   Leiden University

   Affiliation as printed

   Leiden University , The Netherlands

4. Maarten de Rijke

   University of Amsterdam

   Affiliation as printed

   University of Amsterdam , The Netherlands

5. Suzan Verberne Aachen

   Leiden University

   Affiliation as printed

   Leiden University , The Netherlands