Annotator-Centric Active Learning for Subjective NLP Tasks

Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 18537–18555

Abstract

Active Learning (AL) addresses the high costs of collecting human annotations by strategically annotating the most informative samples.However, for subjective NLP tasks, incorporating a wide range of perspectives in the annotation process is crucial to capture the variability in human judgments.We introduce Annotator-Centric Active Learning (ACAL), which incorporates an annotator selection strategy following data sampling.Our objective is two-fold:(1) to efficiently approximate the full diversity of human judgments, and (2) to assess model performance using annotator-centric metrics, which value minority and majority perspectives equally.We experiment with multiple annotator selection strategies across seven subjective NLP tasks, employing both traditional and novel, human-centered evaluation metrics.Our findings indicate that ACAL improves data efficiency and excels in annotator-centric performance evaluations.However, its success depends on the availability of a sufficiently large and diverse pool of annotators to sample from.

Authors 4

1. Michiel van der Meer Aachen Leiden Institute of Advanced Computer Science

   Leiden University · Idiap Research Institute

   Affiliation as printed

   Idiap Research Institute , Switzerland

   Leiden Institute of Advanced Computer Science , Leiden University , The Netherlands

2. Neele Falk

   University of Stuttgart

   Affiliation as printed

   Institute for Natural Language Processing , University of Stuttgart , Germany

3. Pradeep Kumar Murukannaiah

   Delft University of Technology

   Affiliation as printed

   Interactive Intelligence , TU Delft , The Netherlands

4. Enrico Liscio

   Delft University of Technology

   Affiliation as printed

   Interactive Intelligence , TU Delft , The Netherlands