From Co-Creation to Technical Bias Detection Methods: an Interdisciplinary Showcase from the BIAS Project

Leiden Repository (Leiden University)

Abstract

--Pre-Print Version-- Abstract: Societal stereotypes are often reflected in, and can be reinforced by, machine learning models and linguistic resources such as word embeddings. While various benchmarks and bias detection methods have been proposed, most focus exclusively on English. When applied to other languages, these approaches typically rely on direct translations of English resources, overlooking language- and culture-specific nuances. In this paper, we introduce BIAS-WEAT and BIAS-SEAT, two novel metrics designed to detect biases in word embeddings and language models for Dutch, German, Icelandic, Italian, Norwegian and Turkish. Drawing on real-world biases identified through co-creation workshops with native speakers in the context of a hiring situation, we translated these insights into technical evaluation metrics that are applicable to general-purpose language resources. Our interdisciplinary study demonstrates how language models embed and reproduce biases that are specific to their linguistic and geographic contexts, underscoring the need for culturally grounded approaches to bias detection.

Authors 10

1. Mascha Kurpicz-Briki

   Bern University of Applied Sciences

   Affiliation as printed

   Bern University of Applied Sciences

2. Catherine Ikae

   Bern University of Applied Sciences

   Affiliation as printed

   Bern University of Applied Sciences

3. Alexandre Puttick

   Bern University of Applied Sciences

   Affiliation as printed

   Bern University of Applied Sciences

4. Guðbjörg Linda Rafnsdóttir

   University of Iceland

   Affiliation as printed

   University of Iceland

5. Ragna Kemp Haraldsdóttir

   University of Iceland

   Affiliation as printed

   University of Iceland

6. Carlotta Rigotti Aachen

   Leiden University

   Affiliation as printed

   Leiden University

7. Eduard Fosch‐Villaronga Aachen

   Leiden University

   Affiliation as printed

   Leiden University

8. Maria Sangiuliano
   Affiliation as printed

   Smart Venice srl

9. Mark W. Kharas

   Norwegian University of Science and Technology

   Affiliation as printed

   Norwegian University of Science and Technology

10. Roger A. Søraa

    Norwegian University of Science and Technology

    Affiliation as printed

    Norwegian University of Science and Technology