An ELIXIR scoping review on domain-specific evaluation metrics for synthetic data in life sciences

NAR Genomics and Bioinformatics, vol. 8, pp. lqag012

Abstract

Synthetic data (SD) has become an increasingly important asset in the life sciences, helping address data scarcity, privacy concerns, and barriers to data access. Creating artificial datasets that mirror the characteristics of real data allows researchers to develop and validate computational methods in controlled environments. Despite its promise, the adoption of SD in life sciences hinges on rigorous evaluation metrics designed to assess their fidelity and reliability. To explore the current landscape of SD evaluation metrics in distinct life sciences domains, the ELIXIR Machine Learning Focus Group performed a systematic review of the scientific literature following the PRISMA guidelines. Six critical domains were examined to identify current practices for assessing SD. Findings reveal that, while generation methods are rapidly evolving, systematic evaluation is often overlooked, limiting researchers' ability to compare, validate, and trust synthetic datasets across different domains. This systematic review underscores the urgent need for robust, standardized evaluation approaches that not only bolster confidence in SD but also guide its effective and responsible implementation. By laying the groundwork for establishing domain-specific yet interoperable standards, this scoping review paves the way for future initiatives aimed at enhancing the role of SD in scientific discovery, clinical practice and beyond.

Authors 14

1. Styliani-Christina Fragkouli

   National and Kapodistrian University of Athens · Centre for Research and Technology Hellas

   Affiliation as printed

   Department of Biology, National and Kapodistrian University of Athens , Athens 15772 ,

   Institute of Applied Biosciences, Centre for Research & Technology Hellas , Thessaloniki 57001 ,

2. Shaz Iqbal

   University of Edinburgh

   Affiliation as printed

   University of Edinburgh, Usher Institute, Nine BioQuarter , Edinburgh EH16 4UX ,

3. Lisa C. Crossman

   University of East Anglia · Norwich Research Park

   Affiliation as printed

   School of Biological Sciences, University of East Anglia, Norwich Research Park , Norwich NR4 7TJ ,

   SequenceAnalysis.co.uk, Norwich Research Park , Norwich NR4 7UG ,

4. Barbara Gravel

   Université Libre de Bruxelles · Vrije Universiteit Brussel

   Affiliation as printed

   Artificial Intelligence Laboratory, Vrije Universiteit Brussel , Brussel 1050 ,

   Interuniversity Institute of Bioinformatics in Brussels, Université Libre de Bruxelles-Vrije Universiteit Brussel , Brussels 1050 ,

   Machine Learning Group, Université Libre de Bruxelles , Brussel 1050 ,

5. Nagat Masued

   Barcelona Supercomputing Center

   Affiliation as printed

   Barcelona Supercomputing Center (BSC), Plaça Eusebi Güell, 1-3 , Barcelona 08034 ,

6. Mark Onders

   University of Notre Dame

   Affiliation as printed

   Department of Applied and Computational Mathematics and Statistics, University of Notre Dame , Notre Dame, IN 46556 ,

7. Devesh Haseja

   Ollscoil na Gaillimhe – University of Galway

   Affiliation as printed

   School of Mathematical and Statistical Sciences, College of Science and Engineering, University of Galway , Galway, H91 TK33 ,

8. Alex Stikkelman Aachen Department of Human Genetics

   Leiden University Medical Center

   Affiliation as printed

   Department of Human Genetics, Leiden University Medical Center , Leiden 2333 ,

9. Alfonso Valencia

   Institució Catalana de Recerca i Estudis Avançats · Barcelona Supercomputing Center

   Affiliation as printed

   Barcelona Supercomputing Center (BSC), Plaça Eusebi Güell, 1-3 , Barcelona 08034 ,

   ICREA, Pg. Lluís Companys 23 , Barcelona 08010 ,

10. Tom Lenaerts

    Université Libre de Bruxelles · Vrije Universiteit Brussel

    Affiliation as printed

    Artificial Intelligence Laboratory, Vrije Universiteit Brussel , Brussel 1050 ,

    Interuniversity Institute of Bioinformatics in Brussels, Université Libre de Bruxelles-Vrije Universiteit Brussel , Brussels 1050 ,

    Machine Learning Group, Université Libre de Bruxelles , Brussel 1050 ,

11. Fotis E Psomopoulos

    Centre for Research and Technology Hellas

    Affiliation as printed

    Institute of Applied Biosciences, Centre for Research & Technology Hellas , Thessaloniki 57001 ,

12. Pilib Ó Broin

    Ollscoil na Gaillimhe – University of Galway

    Affiliation as printed

    School of Mathematical and Statistical Sciences, College of Science and Engineering, University of Galway , Galway, H91 TK33 ,

13. Núria Queralt-Rosiñach Aachen Department of Human Genetics

    Leiden University Medical Center

    Affiliation as printed

    Department of Human Genetics, Leiden University Medical Center , Leiden 2333 ,

14. Davide Cirillo

    Barcelona Supercomputing Center

    Affiliation as printed

    Barcelona Supercomputing Center (BSC), Plaça Eusebi Güell, 1-3 , Barcelona 08034 ,