Dynamic knowledge graph evaluation: Semantic and syntactic metrics for evaluating changes

Data & Knowledge Engineering, vol. 164, pp. 102611

Abstract

In a world where information is exchanged at an increasing pace, knowledge becomes quickly outdated. Formal constructs that capture human knowledge, such as knowledge graphs and ontologies, need to be updated and evaluated to stay relevant and functioning. However, evaluating knowledge models is labour-intensive and prone to errors. This study addresses the challenge of automatically evaluating changes in existing knowledge graphs. We introduce syntactic and semantic metrics tailored for change evaluation. The metrics are implemented and validated through experiments on knowledge graphs across various domains. In these experiments, real-world changes are simulated by removing concepts and introducing faulty ones before measuring the quality with the syntactic and semantic metrics. The hypothesis is that such changes decrease the quality of the knowledge graph: removing concepts influences syntactic qualities such as the structure of the model, while adding faulty concepts affects semantic qualities like model consistency. The validation results support this hypothesis, demonstrating that the introduced metrics effectively reflect the changes made to the graph. Additionally, the experiments show that the size and domain specialisation of a knowledge graph influence how well the metrics detect changes. Overall, this study proposes a novel set of evaluation metrics and provides evidence of their effectiveness for assessing modifications to knowledge graphs across different domains. These metrics can help developers detect errors, highlight unintended side effects, and flag other quality changes that might otherwise go unnoticed.

Authors 2

1. Roos M. Bakker corresponding Aachen

   Leiden University · Netherlands Organisation for Applied Scientific Research

   Affiliation as printed

   Leiden University Centre for Linguistics, Reuvensplaats 3 - 4, Leiden, 2311 BE, The Netherlands

   TNO, Department Data Science, Anna van Buerenplein 1, Den Haag, 2595 DA, The Netherlands

2. Maaike de Boer

   Netherlands Organisation for Applied Scientific Research

   Affiliation as printed

   TNO, Department Data Science, Anna van Buerenplein 1, Den Haag, 2595 DA, The Netherlands