Multiverse analyses in the classroom

Abstract

Most empirical papers in psychology involve statistical analyses performed on a new or existing dataset. Sometimes the robustness of a finding is demonstrated via data-analytical triangulation (e.g., obtaining comparable outcomes across different operationalizations of the dependent variable), but systematically considering the plethora of alternative analysis pathways is a rarity. In contrast, researchers conducting a multiverse analysis precisely aim to assess the robustness of a finding by methodically examining the seemingly-arbitrary choices pertaining to data processing and/or model building. In the present paper, we describe how the multiverse approach can be successfully implemented in student research projects within psychology programs, drawing on personal experience as instructors. Embedding a multiverse project in students’ curricula addresses an important scientific need, as studies examining the robustness or fragility of phenomena are largely lacking in psychology. More importantly, it offers students an ideal opportunity to put various statistical methods into practice, thereby also raising awareness about the abundance and consequences of arbitrary decisions in data-analytic processing. An attractive practical feature is that one can reuse existing datasets, which proves especially useful when resources are limited, or when circumstances such as the COVID-19 lockdown measures restrict data collection possibilities.

Authors 2

1. Tom Heyman corresponding Aachen Methodology and Statistics Unit Institute of Psychology, Leiden University

   Leiden University

   Affiliation as printed

   Leiden University

   Methodology and Statistics Unit , Institute of Psychology , Leiden University , Wassenaarseweg 52 , 2333 AK Leiden , The Netherlands

2. Wolf Vanpaemel

   KU Leuven

   Affiliation as printed

   KU Leuven