Unified Multi-Scenario Summarization Evaluation and Explanation

IEEE Transactions on Knowledge and Data Engineering, vol. 37, pp. 991–1003

Abstract

Summarization quality evaluation is a non-trivial task in text summarization. Contemporary methods can be mainly categorized into two scenarios: (1)reference-based:evaluating with human-labeled reference summary; (2)reference-free:evaluating the summary consistency of the document. Recent studies mainly focus on one of these scenarios and explore training neural models to align with human criteria and finally give a numeric score. However, the models from different scenarios are optimized individually, which may result in sub-optimal performance since they neglect the shared knowledge across different scenarios. Besides, designing individual models for each scenario caused inconvenience to the user. Moreover, only providing the numeric quality evaluation score for users cannot help users to improve the summarization model, since they do not know why the score is low. Inspired by this, we proposeUnifiedMulti-scenarioSummarizationEvaluator (UMSE) andMulti-AgentSummarizationEvaluationExplainer (MASEE). More specifically, we propose a perturbed prefix tuning method to share cross-scenario knowledge between scenarios and use a self-supervised training paradigm to optimize the model without extra human labeling. Our UMSE is the first unified summarization evaluation framework engaged with the ability to be used in three evaluation scenarios. We propose a multi-agent summary evaluation explanation method MASEE, which employs several LLM-based agents to generate detailed natural language explanations in four different aspects. Experimental results across three typical scenarios on the benchmark dataset SummEval indicate that our UMSE can achieve comparable performance with several existing strong methods that are specifically designed for each scenario. And intensive quantitative and qualitative experiments also demonstrate the effectiveness of our proposed explanation method, which can generate consistent and accurate explanations.

Authors 9

1. Shuo Shang

   University of Electronic Science and Technology of China

   Affiliation as printed

   University of Electronic Science and Technology of China, Chengdu, China

2. Zhitao Yao

   Shandong University

   Affiliation as printed

   Shandong University, Jinan, China

3. Hao Fu

   Shandong University

   Affiliation as printed

   Shandong University, Jinan, China

4. Chongyang Tao

   Beihang University

   Affiliation as printed

   Beihang University, Beijing, China

5. Xiuying Chen

   Mohamed bin Zayed University of Artificial Intelligence

   Affiliation as printed

   Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

6. Feng Wang

   Ant Group (China)

   Affiliation as printed

   Ant Group, Hangzhou, China

7. Yanan Wang

   Ant Group (China)

   Affiliation as printed

   Ant Group, Hangzhou, China

8. Zhaochun Ren Aachen

   Leiden University

   Affiliation as printed

   Leiden University, Leiden, The Netherlands

9. Shen Gao

   University of Electronic Science and Technology of China

   Affiliation as printed

   University of Electronic Science and Technology of China, Chengdu, China