Evaluating the Evaluator: Problems With SemEval-2020 Task 1 for Lexical Semantic Change Detection

Journal of Open Humanities Data, vol. 12

Abstract

This discussion paper re-examines SemEval-2020 Task 1, the most influential shared benchmark for lexical semantic change (LSC) detection, through a three-part evaluative framework: operationalisation, data quality, and benchmark design. First, at the level of operationalisation, the benchmark models semantic change mainly as gain, loss, or redistribution of discrete senses. We argue that while practical for annotation and evaluation, this framing is too narrow to capture gradual, constructional, collocational, and discourse-level change. Also, the gold labels are outcomes of annotation decisions, clustering procedures, and threshold settings, which could potentially limit the validity of the task. Second, at the level of data quality, we show that the benchmark is affected by substantial corpus and preprocessing problems, including Optical Character Recognition (OCR) noise, malformed characters, truncated sentences, inconsistent lemmatisation, Part-of-Speech (POS) tagging errors, and missed targets. These issues can distort model behaviour, complicate linguistic analysis, and reduce reproducibility. Third, at the level of benchmark design, we argue the small curated target sets and limited language coverage reduce realism and increase statistical uncertainty. Taken together, these limitations suggest that the benchmark should be treated as a useful but partial test bed rather than a definitive measure of progress. We therefore call for future datasets and shared tasks to adopt broader theories of semantic change, document preprocessing transparently, expand cross-linguistic coverage, and use more realistic evaluation settings. Such steps are necessary for more valid, interpretable, and generalisable progress in lexical semantic change detection.

Authors 5

1. Bach Phan-Tat corresponding

   KU Leuven

   Affiliation as printed

   Department of Linguistics, KU Leuven, Leuven

2. Kris Heylen Aachen

   Instituut voor de Nederlandse Taal · KU Leuven

   Affiliation as printed

   Department of Linguistics, KU Leuven, Leuven, BE; Instituut voor de Nederlandse Taal, Leiden

3. Dirk Geeraerts

   KU Leuven

   Affiliation as printed

   Department of Linguistics, KU Leuven, Leuven

4. Stefano De Pascale

   Vrije Universiteit Brussel · KU Leuven

   Affiliation as printed

   Department of Linguistics, KU Leuven, Leuven; Department of Linguistics and Literary Studies, Vrije Universiteit Brussel, Brussels

5. Dirk Speelmana

   KU Leuven

   Affiliation as printed

   Department of Linguistics, KU Leuven, Leuven