Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study (Preprint)

Abstract

BACKGROUND Despite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing suitable evaluation approaches in relation to a specific study phase. OBJECTIVE The objective of this project was to provide a structured overview of the existing eHealth evaluation approaches, with the aim of assisting eHealth evaluators in selecting a suitable approach for evaluating their eHealth solution at a specific evaluation study phase. METHODS Three consecutive steps were followed. Step 1 was a systematic scoping review, summarizing existing eHealth evaluation approaches. Step 2 was a concept mapping study asking eHealth researchers about approaches for evaluating eHealth. In step 3, the results of step 1 and 2 were used to develop an “eHealth evaluation cycle” and subsequently compose the online “eHealth methodology guide.” RESULTS The scoping review yielded 57 articles describing 50 unique evaluation approaches. The concept mapping study questioned 43 eHealth researchers, resulting in 48 unique approaches. After removing duplicates, 75 unique evaluation approaches remained. Thereafter, an “eHealth evaluation cycle” was developed, consisting of six evaluation study phases: conceptual and planning, design, development and usability, pilot (feasibility), effectiveness (impact), uptake (implementation), and all phases. Finally, the “eHealth methodology guide” was composed by assigning the 75 evaluation approaches to the specific study phases of the “eHealth evaluation cycle.” CONCLUSIONS Seventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online “eHealth methodology guide.” By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the “eHealth evaluation cycle,” the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions.

Authors 11

1. Tobias Nicolaas Bonten corresponding Aachen Department of Public Health and Primary Care (PHEG)

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands

   National eHealth Living Lab, Leiden, Netherlands

2. Anneloek Rauwerdink

   Amsterdam University Medical Centers

   Affiliation as printed

   Department of Surgery, Amsterdam Gastroenterology and Metabolism, Amsterdam UMC, Amsterdam, Netherlands

3. Jeremy C Wyatt

   University of Southampton · Wessex Institute of Technology

   Affiliation as printed

   Wessex Institute, University of Southampton, Southampton, United Kingdom

4. Marise Jeannine Kasteleyn Aachen Department of Public Health and Primary Care (PHEG)

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands

   National eHealth Living Lab, Leiden, Netherlands

5. Leonard Witkamp Aachen Department of Public Health and Primary Care Leiden University Medical Centre

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Public Health & Primary Care, Room V6-22 PO Box 9600 Leiden, 2300 RC Netherlands

   Department of Public Health and Primary Care Leiden University Medical Centre

   Research Group

6. Heleen M. Riper

   Vrije Universiteit Amsterdam

   Affiliation as printed

   Department of Clinical, Neuro and Developmental Psychology, Vrije Universiteit, Amsterdam, Netherlands

7. Julia E.W.C. van Gemert‐Pijnen

   University of Twente

   Affiliation as printed

   Department of Psychology, Health and Technology, Centre for eHealth and Wellbeing Research, University of Twente, Enschede, Netherlands

8. Kathrin M. Cresswell

   University of Edinburgh

   Affiliation as printed

   Centre of Medical Informatics, Usher Institute, The University of Edinburgh, Medical School, Edinburgh, United Kingdom

9. Aziz Sheikh

   University of Edinburgh

   Affiliation as printed

   Centre of Medical Informatics, Usher Institute, The University of Edinburgh, Medical School, Edinburgh, United Kingdom

10. Marlies P. Schijven

    Amsterdam University Medical Centers

    Affiliation as printed

    Department of Surgery, Amsterdam Gastroenterology and Metabolism, Amsterdam UMC, Amsterdam, Netherlands

11. Niels Henrik Chavannes Aachen Department of Public Health and Primary Care (PHEG)

    Leiden University · Leiden University Medical Center

    Affiliation as printed

    Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands

    National eHealth Living Lab, Leiden, Netherlands