Enhancing Training Data Quality through Influence Scores for Generalizable Classification: A Case Study on Sexism Detection

Underline Science Inc.

Abstract

The quality of training data is crucial for the performance of supervised machine learning models. In particular, poor annotation quality and spurious correlations between labels and features in text dataset can significantly degrade model generalization. This problem is especially pronounced in harmful language detection, where prior studies have revealed major deficiencies in existing datasets. In this work, we design and test data selection methods based on learnability measures to improve dataset quality. Using a sexism dataset with counterfactuals designed to avoid spurious correlations, we show that pruning with EL2N and PVI scores can lead to significant performance increases and outperforms submodular and random selection. Our analysis reveals that in presence of label imbalance models rely on dataset shortcuts; especially easy-to-classify sexist instances and hard-to-classify non-sexist instances contain shortcuts. Pruning these instances leads to performances increases. Pruning hard-to-classify instances is in general a promising strategy as well when shortcuts are not present.

Authors 5

1. Association for Computational Linguistics 2025
2. Jose Maria Alonso

   University of Southern California

   Affiliation as printed

   University of Southern California

3. Dennis Assenmacher

   GESIS - Leibniz Institute for the Social Sciences

   Affiliation as printed

   GESIS

4. Rabiraj Bandyopadhyay

   GESIS - Leibniz Institute for the Social Sciences

   Affiliation as printed

   GESIS-Leibniz Institute for the Social Sciences

5. Claudia Wagner Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen