Mind the Gap: Exposing LLM Translation Blind Spots Using the AlphaMWE Multilingual Parallel Corpus

arXiv (Cornell University)

Abstract

LLMs' performance on machine translation (MT) tasks is often dependent on the data availability in the specific domains and language pairs that they are trained upon. To examine if Multiword Expressions (MWEs) still set a bottleneck for LLMs regarding language understanding and translation, we report the system performances from the WMT2026 Test Suites shared task, for which we used the publicly available multilingual parallel corpus AlphaMWE as the test suites. We received 31 MT systems' outputs covering English to Chinese (zh), Polish (pl), German (de), Arabic (ar) including Modern Standard Arabic (MSA) and two dialectal ones (Egyptian and Tunisian Arabic). We carried out automatic evaluations using BLEU, ChrF, BERT-score to select the Top3 systems per language pair, followed up with human evaluations on the selected systems. Our findings show that: figurative/MWE phenomena remain challenging; automatic metrics sometimes disagree; human evaluation uncovers language-specific errors hidden by aggregate scores; human evaluation re-ranks the top-3 systems.

Authors 9

1. L. Han Aachen

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Leiden University

   Leiden University Medical Centre

2. Jiahui Liang Aachen

   Leiden University

   Affiliation as printed

   Leiden University

3. A. Latusek

   Institute of Computer Science

   Affiliation as printed

   Institute of Computer Science PAS , Warsaw

4. K. El Haff

   University of Ljubljana

   Affiliation as printed

   University of Ljubljana

5. A. Haddad

   Universidad de Granada

   Affiliation as printed

   University of Granada

6. J. Höfgen

   Technical University of Munich

   Affiliation as printed

   Technical University of Munich

7. K. Evang

   Heinrich Heine University Düsseldorf

   Affiliation as printed

   Heinrich Heine University Düsseldorf

8. M. Ma

   Google DeepMind (United Kingdom)

   Affiliation as printed

   Google DeepMind

9. M. Zhyrko Aachen

   Leiden University

   Affiliation as printed

   Leiden University