Identifying careless responding in ecological momentary assessment: Inconsistent signals from different detection methods in the WARN-D Data.

Psychological Methods

Abstract

Collecting intensive longitudinal data via methods like ecological momentary assessment (EMA) is necessary to study dynamics in psychological constructs.However, detecting careless responding is more important than ever, as this methodology burdens participants more than traditional single-survey studies.If undetected, careless responding reduces data quality and can lead to invalid findings by distorting correlations between items and constructs and compromising the psychometric properties of scales.In turn, this hinders progress toward cumulative psychological science.In recent years, several data-driven approaches have been proposed that assess whether an individual was attentive or careless for each time point-specific measurement after data collection.However, the variety and complexity of available methods can be overwhelming for researchers.In this article, we summarized, applied, and compared 13 methods that flag careless responses in a rich EMA data set from the WARN-D study with 206,318 observations (from n 5 1,194 participants).Together with our detailed online supplementary tutorials, the article serves as a guide for researchers to evaluate and report careless responding in their own data.At the same time, we show that the development of methods for careless responding detection is still in its early stages, as the studied approaches produced very different results.We provide recommendations for which methods are currently most promising based on our findings and discuss important next steps in validating them empirically.

Authors 12

1. Leonie V. D. E. Vogelsmeier

   Tilburg University

   Affiliation as printed

   Department of Methodology and Statistics, Tilburg University

2. Björn Sebastian Siepe

   Philipps University of Marburg

   Affiliation as printed

   Psychological Methods Lab, Department of Psychology, University of Marburg

3. Esther Ulitzsch

   University of Oslo

   Affiliation as printed

   Centre for Educational Measurement, University of Oslo

4. Gudrun Vera Eisele

   KU Leuven

   Affiliation as printed

   Center for Contextual Psychiatry, Department of Neuroscience, KU Leuven

   KU Leuven

5. Joran Geeraerts

   Vrije Universiteit Brussel

   Affiliation as printed

   Faculty of Medicine and Pharmacy, Vrije Universiteit Brussel

6. Adam Klocek

   Masaryk University

   Affiliation as printed

   Psychology Research Institute, Faculty of Social Studies, Masaryk University

7. Ricarda K. K. Proppert Aachen Department of Clinical Psychology

   Leiden University

   Affiliation as printed

   Department of Clinical Psychology, Leiden University

8. Carlotta Rieble Aachen Department of Clinical Psychology

   Leiden University

   Affiliation as printed

   Department of Clinical Psychology, Leiden University

9. Rayyan Tutunji Aachen Department of Clinical Psychology

   Leiden University

   Affiliation as printed

   Department of Clinical Psychology, Leiden University

10. Egon Dejonckheere

    Tilburg University

    Affiliation as printed

    Department of Medical and Clinical Psychology, Tilburg University

11. Peter Kuppens

    KU Leuven

    Affiliation as printed

    Department of Psychology, University of Leuven

12. Eiko I. Fried Aachen Department of Clinical Psychology

    Leiden University

    Affiliation as printed

    Department of Clinical Psychology, Leiden University