Subjective evidence evaluation survey for many-analysts studies
Royal Society Open Science, vol. 11, pp. 240125
Abstract
Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
Authors 40
-
Alexandra Sarafoglou corresponding
Affiliation as printed
Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
-
Affiliation as printed
Utrecht University, Utrecht, The Netherlands
-
Affiliation as printed
Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
-
Affiliation as printed
Institute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary
-
Affiliation as printed
Heymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands
-
Affiliation as printed
Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA
-
Dartmouth College · Hebrew University of Jerusalem
Affiliation as printed
Dartmouth College, Hanover, NH, USA
Hebrew University of Jerusalem, Jerusalem, Israel
-
Affiliation as printed
Institute for Psychology, University of Münster, Münster, Germany
-
Harvard University · McLean Hospital
Affiliation as printed
Center for Depression, Anxiety and Stress Research, McLean Hospital, Belmont, MA, USA
Department of Psychiatry, Harvard Medical School, Boston, MA, USA
-
Affiliation as printed
Department of Business, University of Idaho, Moscow, ID, USA
-
Affiliation as printed
Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
-
Stockholm School of Economics · Universität Innsbruck
Affiliation as printed
Stockholm School of Economics, Stockholm, Sweden
University of Innsbruck, Innsbruck, Tirol, Austria
-
Affiliation as printed
Department of Psychology, Leiden University, Leiden, The Netherlands
-
Affiliation as printed
Nieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands
Nieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands AS, 0000-0003
-
Ludwig-Maximilians-Universität München
Affiliation as printed
Department of Statistics, Ludwig-Maximilians-Universität München, Munchen, Bayern, Germany
-
Affiliation as printed
University of Innsbruck, Innsbruck, Tirol, Austria
-
Affiliation as printed
University of Innsbruck, Innsbruck, Tirol, Austria
-
Affiliation as printed
Seattle University, Seattle, WA, USA
-
Stanford Medicine · Stanford University
Affiliation as printed
Meta-Research Innovation Center at Stanford (METRICS) and Departments of Medicine, of Epidemiology and of Population Health, of Biomedical Data Science, and of Statistics, Stanford University, Stanford, CA, USA
-
Affiliation as printed
Stockholm School of Economics, Stockholm, Sweden
-
Affiliation as printed
University of Innsbruck, Innsbruck, Tirol, Austria
-
Affiliation as printed
University of Conneticut, Storrs, CT, USA
-
Commissariat à l'Énergie Atomique et aux Énergies Alternatives · Université Paris-Saclay
Affiliation as printed
Neurospin CEA, Gif-sur-Yvette, Île-de-France, France
University Paris-Saclay, Gif-sur-Yvette, France
-
Affiliation as printed
Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
-
Affiliation as printed
Vrije Universiteit Amsterdam, Amsterdam, Noord-Holland, The Netherlands
-
Affiliation as printed
Karolinska Institutet, Solna, Sweden
-
Affiliation as printed
Heymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands
-
European School of Management and Technology
Affiliation as printed
ESMT Berlin, Berlin, Germany
-
Institut für Medizinische Informationsverarbeitung, Biometrie und Epidemiologie · Ludwig-Maximilians-Universität München
Affiliation as printed
Department of Statistics and The Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Munich, Munchen, Bayern, Germany
The Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Munich, Munchen, Bayern, Germany
-
Affiliation as printed
Division of Psychology and Language Sciences, University College London, 26 Bedford Way, London WC1H 0AP, UK
-
University of Illinois Urbana-Champaign
Affiliation as printed
University of Illinois—Urbana-Champaign, Urbana, IL, USA
University of Illinois-Urbana-Champaign, Urbana, IL, USA
-
Affiliation as printed
School of Law, University of Virginia, 580 Massie Road, Charlottesville, VA, USA
-
Affiliation as printed
Nieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands
Nieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands AS, 0000-0003
-
Affiliation as printed
Institute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary
-
Max Planck Institute for Empirical Aesthetics
Affiliation as printed
Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany
-
Affiliation as printed
University of Leuven, Leuven, Belgium
-
Affiliation as printed
INSEAD, Fontainebleau, Île-de-France, France
-
Affiliation as printed
University of Leuven, Leuven, Belgium
-
Affiliation as printed
Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
-
Affiliation as printed
Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
Cited by 5 stored of 5
5 results
No patents citing this paper on Lens.org (checked 2026-10-11).