Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study

Journal of Medical Internet Research, vol. 22, pp. e17774

Abstract

BACKGROUND: Despite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing suitable evaluation approaches in relation to a specific study phase. OBJECTIVE: The objective of this project was to provide a structured overview of the existing eHealth evaluation approaches, with the aim of assisting eHealth evaluators in selecting a suitable approach for evaluating their eHealth solution at a specific evaluation study phase. METHODS: Three consecutive steps were followed. Step 1 was a systematic scoping review, summarizing existing eHealth evaluation approaches. Step 2 was a concept mapping study asking eHealth researchers about approaches for evaluating eHealth. In step 3, the results of step 1 and 2 were used to develop an "eHealth evaluation cycle" and subsequently compose the online "eHealth methodology guide." RESULTS: The scoping review yielded 57 articles describing 50 unique evaluation approaches. The concept mapping study questioned 43 eHealth researchers, resulting in 48 unique approaches. After removing duplicates, 75 unique evaluation approaches remained. Thereafter, an "eHealth evaluation cycle" was developed, consisting of six evaluation study phases: conceptual and planning, design, development and usability, pilot (feasibility), effectiveness (impact), uptake (implementation), and all phases. Finally, the "eHealth methodology guide" was composed by assigning the 75 evaluation approaches to the specific study phases of the "eHealth evaluation cycle." CONCLUSIONS: Seventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online "eHealth methodology guide." By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the "eHealth evaluation cycle," the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions.

Authors 11

1. Tobias Nicolaas Bonten Aachen Department of Public Health and Primary Care (PHEG)

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands

   National eHealth Living Lab, Leiden, Netherlands

   Leiden University Medical Center

2. Anneloek Rauwerdink

   Amsterdam University Medical Centers

   Affiliation as printed

   Department of Surgery, Amsterdam Gastroenterology and Metabolism, Amsterdam UMC, Amsterdam, Netherlands

   Amsterdam UMC

3. Jeremy C Wyatt

   University of Southampton · Wessex Institute of Technology

   Affiliation as printed

   Wessex Institute, University of Southampton, Southampton, United Kingdom

   University of Southampton *

4. Marise Jeannine Kasteleyn Aachen Department of Public Health and Primary Care (PHEG)

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands

   National eHealth Living Lab, Leiden, Netherlands

   Leiden University Medical Center

5. Leonard Witkamp

   KSYOS TeleMedical Centre · Amsterdam University Medical Centers

   Affiliation as printed

   Department of Medical Informatics, Amsterdam UMC, Amsterdam, Netherlands

   Ksyos Health Management Research, Amstelveen, Netherlands

   Amsterdam UMC

6. Heleen M. Riper

   Vrije Universiteit Amsterdam

   Affiliation as printed

   Department of Clinical, Neuro and Developmental Psychology, Vrije Universiteit, Amsterdam, Netherlands

   Vrije Universiteit;

7. Julia E.W.C. van Gemert‐Pijnen

   University of Twente

   Affiliation as printed

   Department of Psychology, Health and Technology, Centre for eHealth and Wellbeing Research, University of Twente, Enschede, Netherlands

   Psychology, Health & Technology

8. Kathrin M. Cresswell

   University of Edinburgh

   Affiliation as printed

   Centre of Medical Informatics, Usher Institute, The University of Edinburgh, Medical School, Edinburgh, United Kingdom

   *University of Edinburgh Medical School

9. Aziz Sheikh

   University of Edinburgh

   Affiliation as printed

   Centre of Medical Informatics, Usher Institute, The University of Edinburgh, Medical School, Edinburgh, United Kingdom

   *University of Edinburgh Medical School

10. Marlies P. Schijven

    Amsterdam University Medical Centers

    Affiliation as printed

    Department of Surgery, Amsterdam Gastroenterology and Metabolism, Amsterdam UMC, Amsterdam, Netherlands

    Amsterdam UMC

11. Niels Henrik Chavannes Aachen Department of Public Health and Primary Care (PHEG)

    Leiden University · Leiden University Medical Center

    Affiliation as printed

    Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands

    National eHealth Living Lab, Leiden, Netherlands

    Leiden University Medical Center