Understanding validity criteria in technology-enhanced learning: A systematic literature review

Computers & Education, vol. 220, pp. 105128

Abstract

Technological aids are ubiquitous in today's educational environments. Whereas much of the dust has settled in the debate on how to validate traditional educational solutions, in the area of technology-enhanced learning (TEL) many questions still remain. Technologies often abstract away student behaviour by condensing actions into numbers, meaning teachers have to assess student data rather than observing students directly. With the rapid adoption of artificial intelligence in education, it is timely to obtain a clear image of the landscape of validity criteria relevant to TEL. In this paper, we conduct a systematic review of research on TEL interventions, where we combine active learning for title and abstract screening with a backward snowballing phase. We extract information on the validity criteria used to evaluate TEL solutions, along with the methods employed to measure these criteria. By combining data on the research methods (qualitative versus quantitative) and knowledge source (theory versus practice) used to inform validity criteria, we ground our results epistemologically. We find that validity criteria tend to be assessed more positively when quantitative methods are used and that validation framework usage is both rare and fragmented. Yet, we also find that the prevalence of different validity criteria and the research methods used to assess them are relatively stable over time, implying that a strong foundation exists to design holistic validation frameworks with the potential to become commonplace in TEL research.

Authors 4

1. Max van Haastrecht corresponding Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Einsteinweg 55, Leiden, 2333, CC, the Netherlands

2. M. Haas Aachen Department of Public Health and Primary Care

   Leiden University Medical Center

   Affiliation as printed

   Department of Public Health and Primary Care, Leiden University Medical Center (LUMC), Albinusdreef 2, Leiden, 2333, ZA, the Netherlands

3. Matthieu J. S. Brinkhuis

   Utrecht University

   Affiliation as printed

   Department of Information and Computing Sciences, Utrecht University, Princetonplein 5, Utrecht, 3584, CC, the Netherlands

4. Marco René Spruit Aachen Department of Public Health and Primary Care Leiden Institute of Advanced Computer Science

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Public Health and Primary Care, Leiden University Medical Center (LUMC), Albinusdreef 2, Leiden, 2333, ZA, the Netherlands

   Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Einsteinweg 55, Leiden, 2333, CC, the Netherlands