A Sustainable Data Extraction Documentation Process for Literature Studies

Abstract

Background.Open Science advocates for transparency of research outcomes; however, software engineering (SE) research often lacks adequate availability of essential artifacts. This deficiency hinders reproducibility, sustainability, and the broader adoption of Open Science practices. Evidence-Based Software Engineering (EBSE) emphasizes the systematic use of empirical evidence to evaluate SE practices. Systematic literature studies form a cornerstone of EBSE but face challenges in persistency, particularly for data extraction results in a multi-reviewer context, due to limited machine-readable formats and inefficiencies in reviewer collaboration. Objective. This paper introduces a practical guideline and documentation process for conducting systematic literature studies with a categorical classification objective (e.g., scoping studies). The approach extends existing methodologies by integrating tool support in terms of version control and machine-readable formats, thereby reinforcing Open Science principles. Method. We performed a feature analysis, following the framework of Marshall et al., to evaluate the suitability of selected tools for supporting documentation and coordination in multi-reviewer data extraction scenarios. Results. Our proposal for a tool-supported data extraction achieves a high score in the feature analysis, indicating suitability and enhancement of common practices. Our guideline improves the documentation of data extraction results in literature studies and addresses key challenges related to traceability, consistency, and reviewer coordination. Conclusion. Research artifacts are critical for the sustainability of SE research and the advancement of Open Science. Our proposed guideline provides a structured, tool-supported approach that enhances transparency, reproducibility, and efficiency in EBSE studies.

Authors 5

1. Angelika Kaplan

   Karlsruhe Institute of Technology

   Affiliation as printed

   Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

2. Marco Konersmann Aachen Software Engineering

   RWTH Aachen University

   Affiliation as printed

   Software Engineering, RWTH Aachen University, Aachen, Germany

3. Thomas Kühn

   Martin Luther University Halle-Wittenberg

   Affiliation as printed

   Martin-Luther-Universität Halle-Wittenberg, Halle (Saale), Germany

4. Raffaela Mirandola

   Karlsruhe Institute of Technology

   Affiliation as printed

   Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

5. Ralf Reussner

   Karlsruhe Institute of Technology

   Affiliation as printed

   Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany