SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing

Frontiers in Research Metrics and Analytics, vol. 6, pp. 685591

Abstract

Research output has grown significantly in recent years, often making it difficult to see the forest for the trees. Systematic reviews are the natural scientific tool to provide clarity in these situations. However, they are protracted processes that require expertise to execute. These are problematic characteristics in a constantly changing environment. To solve these challenges, we introduce an innovative systematic review methodology: SYMBALS. SYMBALS blends the traditional method of backward snowballing with the machine learning method of active learning. We applied our methodology in a case study, demonstrating its ability to swiftly yield broad research coverage. We proved the validity of our method using a replication study, where SYMBALS was shown to accelerate title and abstract screening by a factor of 6. Additionally, four benchmarking experiments demonstrated the ability of our methodology to outperform the state-of-the-art systematic review methodology FAST2.

Authors 5

1. Max van Haastrecht corresponding

   Utrecht University

   Affiliation as printed

   Department of Information and Computing Sciences, Utrecht University, Utrecht, Netherlands

2. Injy Sarhan

   Utrecht University · Arab Academy for Science, Technology, and Maritime Transport

   Affiliation as printed

   Department of Computer Engineering, Arab Academy for Science, Technology and Maritime Transport (AASTMT), Alexandria, Egypt

   Department of Information and Computing Sciences, Utrecht University, Utrecht, Netherlands

3. Bilge Yiğit Özkan

   Utrecht University

   Affiliation as printed

   Department of Information and Computing Sciences, Utrecht University, Utrecht, Netherlands

4. Matthieu J. S. Brinkhuis

   Utrecht University

   Affiliation as printed

   Department of Information and Computing Sciences, Utrecht University, Utrecht, Netherlands

5. Marco René Spruit Aachen Department of Public Health and Primary Care Leiden Institute of Advanced Computer Science

   Leiden University · Utrecht University · Leiden University Medical Center

   Affiliation as printed

   Department of Information and Computing Sciences, Utrecht University, Utrecht, Netherlands

   Department of Public Health and Primary Care, Leiden University Medical Center (LUMC), Leiden, Netherlands

   Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Leiden, Netherlands