Crowdsourcing multiverse analyses to explore the impact of different data-processing and analysis decisions: A tutorial.
Abstract
When processing and analyzing empirical data, researchers regularly face choices that may appear arbitrary (e.g., how to define and handle outliers). If one chooses to exclusively focus on a particular option and conduct a single analysis, its outcome might be of limited utility. That is, one remains agnostic regarding the generalizability of the results, because plausible alternative paths remain unexplored. A multiverse analysis offers a solution to this issue by exploring the various choices pertaining to data-processing and/or model building, and examining their impact on the conclusion of a study. However, even though multiverse analyses are arguably less susceptible to biases compared to the typical single-pathway approach, it is still possible to selectively add or omit pathways. To address this issue, we outline a novel, more principled approach to conducting multiverse analyses through crowdsourcing. The approach is detailed in a step-by-step tutorial to facilitate its implementation. We also provide a worked-out illustration featuring the Semantic Priming Across Many Languages project, thereby demonstrating its feasibility and its ability to increase objectivity and transparency. (PsycInfo Database Record (c) 2026 APA, all rights reserved).
Authors 63
-
Affiliation as printed
Methodology and Statistics Unit, Institute of Psychology, Leiden University
-
Affiliation as printed
Department of Cognition, Emotion, and Methods in Psychology, University of Vienna
-
Affiliation as printed
Department of Psychology, University of Alabama
-
Affiliation as printed
King's Business School, King's College London
-
Slovak Academy of Sciences · Institute of Social Sciences of the Centre of Social and Psychological Sciences of the Slovak Academy of Sciences
Affiliation as printed
Institute of Social Sciences, CSPS, Slovak Academy of Sciences
-
Affiliation as printed
Department of Neuroscience, Padova Neuroscience Center, University of Padova
-
Affiliation as printed
Faculty of Arts, Psychology and Theology, Abo Akademi University
-
Affiliation as printed
Institute of Psychology, Faculty of Philosophy, Jagiellonian University
-
Affiliation as printed
Department of Food, Pharma, and Healthcare, Saint Joseph's University
-
Affiliation as printed
Department of Psychology, Franklin and Marshall College
-
Affiliation as printed
Department of Psychology and Neuroscience, Nova Southeastern University
-
Université Paris Nanterre · Laboratoire Parisien de Psychologie Sociale
Affiliation as printed
Laboratoire Parisien de Psychologie Sociale, Universite Paris Nanterre
-
Johannes Kepler University of Linz
Affiliation as printed
Department of Economic Psychology, Johannes Kepler University Linz
-
Affiliation as printed
Department of Psychology and Neuroscience, Nova Southeastern University
-
Affiliation as printed
Department of Psychology, Faculty of Philosophy, University of Belgrade
-
Ollscoil na Gaillimhe – University of Galway
Affiliation as printed
School of Psychology, University of Galway
-
Affiliation as printed
Department of Psychology, Sapienza University of Rome
-
Affiliation as printed
Department of Dynamic and Clinical Psychology and Health Studies, Sapienza University of Rome
-
Affiliation as printed
Department of Psychology, University of Concepcion
-
FOM University of Applied Sciences for Economics and Management
Affiliation as printed
FOM University of Applied Sciences
-
California State University, San Bernardino
Affiliation as printed
California State University-San Bernardino
-
Affiliation as printed
Laboratoire de Psychologie Sociale, Universite Paris Cite
-
University of Missouri–St. Louis
Affiliation as printed
University of Missouri-St. Louis
-
York College, City University of New York
Affiliation as printed
York College, City University of New York
-
University of Pavol Jozef Šafárik
Affiliation as printed
Department of Psychology, Faculty of Arts, Pavol Jozef Safarik University in Kosice
-
Affiliation as printed
Department of Psychology, George Fox University
-
Affiliation as printed
IDN Being Human, Institute of Psychology, University of Wroclaw
-
Affiliation as printed
Department of Psychology, University of Milano-Bicocca
-
Affiliation as printed
College of Education, Psychology, and Social Work, Flinders University
-
Affiliation as printed
Department of Psychology, University of Navarra
-
Affiliation as printed
Department of Education, Literatures, Intercultural Studies, Languages and Psychology, University of Florence
-
Affiliation as printed
Department of Psychology, Midwestern State University
-
Affiliation as printed
Department of Developmental Psychology and Socialisation, University of Padova
-
Ben-Gurion University of the Negev
Affiliation as printed
Department of Management, Ben-Gurion University of the Negev
-
University of Nebraska–Lincoln
Affiliation as printed
Department of Psychology, Center for Brain, Biology, and Behavior, University of Nebraska-Lincoln
-
Affiliation as printed
School of Communication and Culture, Aarhus University
-
MSH Medical School Hamburg – University of Applied Sciences and Medical University
Affiliation as printed
Department of Psychology, Medical School Hamburg (MSH)
-
Affiliation as printed
Department of Comparative and General Linguistics, Faculty of Arts, University of Ljubljana
-
National and Kapodistrian University of Athens
Affiliation as printed
Department of Primary Education, National and Kapodistrian University of Athens
-
Affiliation as printed
Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London
-
University of Silesia in Katowice
Affiliation as printed
Institute of Psychology, University of Silesia in Katowice
-
Affiliation as printed
Institute of Medical Psychology and Behavioral Neurobiology, University of Tuebingen
-
Affiliation as printed
Department of Methodology, Universitat de Valencia
-
Norwegian University of Science and Technology
Affiliation as printed
Department of Psychology, Norwegian University of Science and Technology
-
Affiliation as printed
Institute of Psychology, RWTH Aachen University
-
Affiliation as printed
Department of Psychology and Psychotherapy, Witten/Herdecke University
-
Affiliation as printed
Department of Linguistics and Scandinavian Studies, University of Oslo
-
Brandenburg University of Technology Cottbus-Senftenberg
Affiliation as printed
Department of Psychology, Brandenburg University of Technology Cottbus-Senftenberg
-
Affiliation as printed
Department of Psychology, Ashland University
-
National University of Singapore
Affiliation as printed
Department of Psychology, Faculty of Arts and Social Sciences, National University of Singapore
-
Affiliation as printed
Department of Psychology, University of Oslo
-
Affiliation as printed
Department of Psychology, Goethe University Frankfurt
-
Université du Québec à Montréal
Affiliation as printed
Department of Psychology, Universite du Quebec a Montreal
-
Affiliation as printed
Programa de Investigacion Asociativa (PIA) en Ciencias Cognitivas, Centro de Investigacion en Ciencias Cognitivas (CICC), Facultad de Psicologia, Universidad de Talca
-
Universidad Autónoma de Madrid
Affiliation as printed
Department of Basic Psychology, Universidad Autonoma de Madrid
-
Affiliation as printed
Department of Psychology, Arizona State University
-
Affiliation as printed
Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London
-
Affiliation as printed
Department of Psychology, Education and Child Studies, Erasmus University Rotterdam
-
Affiliation as printed
Department of Developmental Psychology and Socialisation, University of Padova
-
Leuphana University of Lüneburg
Affiliation as printed
Institute for Sustainability Psychology, Leuphana University of Luneburg
-
Affiliation as printed
Faculty of Arts and Science, Kyushu University
-
Hong Kong University of Science and Technology
Affiliation as printed
Division of Social Science, Hong Kong University of Science and Technology
-
Harrisburg University of Science and Technology
Affiliation as printed
Analytics Program, Harrisburg University of Science and Technology
Cited by 2 stored of 2
2 results
No patents citing this paper on Lens.org (checked 2026-10-06).