Ara-HOPE: Human-Centric Post-Editing Evaluation for Dialectal Arabic to Modern Standard Arabic Translation

Underline Science Inc.

Abstract

This paper addresses key gaps in assessing Dialectal Arabic to Modern Standard Arabic (DA-MSA) translation quality, where standard automatic metrics and general-purpose human evaluation methods fail to capture dialect-specific errors. We introduce Ara-HOPE, a human-centric post-editing evaluation framework comprising a five-category error taxonomy and a decision-tree annotation protocol designed for consistent use by minimally trained native dialect speakers. Using Ara-HOPE, we comparatively evaluate three mainstream systems (Jais, GPT-3.5, and NLLB-200) to quantify error patterns, pinpoint systematic weaknesses, and provide actionable guidance for improving dialect-aware MT. Our analysis shows clear performance differences across systems and highlights dialect-specific terminology handling and semantic preservation as the most persistent challenges in DA-MSA translation.

Authors 4

1. Association for Computational Linguistics 2026
2. Abdullah Alabdullah

   University of Edinburgh

   Affiliation as printed

   The University of Edinburgh

3. Lifeng Han Aachen

   Leiden University

   Affiliation as printed

   Leiden University

4. Chenghua Lin

   University of Manchester

   Affiliation as printed

   University of Manchester