Subjective Evidence Evaluation Survey For Many-Analysts Studies
Abstract
Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same data set by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g., effect size) provided by each analysis team. Although informative about the range of plausible effects in a data set, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item Subjective Evidence Evaluation Survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
Authors 40
-
Affiliation as printed
University of Amsterdam
-
Affiliation as printed
Utrecht University
-
Affiliation as printed
University of Amsterdam
-
Affiliation as printed
Institute of Psychology , ELTE Eötvös Lorénd University , Budapest , Hungary
Institute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary
-
Affiliation as printed
Heymans Institute for Psychological Research ,
Heymans Institute for Psychological Research,
University of Groningen
-
Affiliation as printed
Allen School of Computer Science & Engineering , University of Washington
Allen School of Computer Science & Engineering, University of Washington
-
Dartmouth College · Hebrew University of Jerusalem
Affiliation as printed
Dartmouth College
Hebrew University of Jerusalem
-
Affiliation as printed
Institute for Psychology , University of Münster , Germany
Institute for Psychology, University of Münster, Germany
-
Harvard University · McLean Hospital
Affiliation as printed
Center for Depression, Anxiety and Stress Research , McLean Hospital , Belmont , MA
Department of Psychiatry , Harvard Medical School , Boston , MA
Center for Depression, Anxiety and Stress Research, McLean Hospital, Belmont, MA
Department of Psychiatry, Harvard Medical School, Boston, MA
-
Affiliation as printed
University of Idaho
-
Affiliation as printed
University of Amsterdam
-
Stockholm School of Economics · Universität Innsbruck
Affiliation as printed
Stockholm School of Economics
University of Innsbruck
-
Affiliation as printed
Department of Psychology , Leiden University
Department of Psychology, Leiden University
-
Affiliation as printed
Nieuwenhuis Institute for Educational Research , University of Groningen
Nieuwenhuis Institute for Educational Research, University of Groningen
-
Ludwig-Maximilians-Universität München
Affiliation as printed
Department of Statistics Ludwig-Maximilians-Universität München
-
Affiliation as printed
University of Innsbruck
-
Affiliation as printed
University of Innsbruck
-
Affiliation as printed
Seattle University
-
Stanford Medicine · Stanford University
Affiliation as printed
Meta-Research Innovation Center at Stanford (METRICS) and
Meta-Research Innovation Center at Stanford (METRICS) and Departments of Medicine, of Epidemiology and of Population Health, of Biomedical Data Science, and of Statistics, Stanford University, Stanford, USA
-
Affiliation as printed
Stockholm School of Economics
-
Affiliation as printed
University of Innsbruck
-
Affiliation as printed
University of Conneticut
-
Commissariat à l'Énergie Atomique et aux Énergies Alternatives · Université Paris-Saclay
Affiliation as printed
Neurospin CEA
University Paris-Saclay
-
Affiliation as printed
University of Amsterdam
-
Affiliation as printed
Vrije Universiteit Amsterdam
-
Affiliation as printed
Karolinska Institutet
-
Affiliation as printed
Heymans Institute for Psychological Research ,
Heymans Institute for Psychological Research,
University of Groningen
-
European School of Management and Technology
Affiliation as printed
ESMT Berlin
-
Institut für Medizinische Informationsverarbeitung, Biometrie und Epidemiologie · Ludwig-Maximilians-Universität München
Affiliation as printed
Department of Statistics LMU Munich
The Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Munich
-
Affiliation as printed
University College London
-
University of Illinois Urbana-Champaign
Affiliation as printed
University of Illinois -Urbana-Champaign
-
Affiliation as printed
University of Virginia
-
Affiliation as printed
Nieuwenhuis Institute for Educational Research , University of Groningen
Nieuwenhuis Institute for Educational Research, University of Groningen
-
Affiliation as printed
Institute of Psychology , ELTE Eötvös Lorénd University , Budapest , Hungary
Institute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary
-
Max Planck Institute for Empirical Aesthetics
Affiliation as printed
Max Planck Institute for Empirical Aesthetics , Germany
Max Planck Institute for Empirical Aesthetics, Germany
-
Affiliation as printed
University of Leuven , Belgium
University of Leuven, Belgium
-
Affiliation as printed
INSEAD
-
Affiliation as printed
KU Leuven
University of Leuven, Belgium
-
University of Groningen · Tilburg University
Affiliation as printed
Department of Methodology and Statistics , Tilburg University
University of Groningen
Department of Methodology and Statistics, Tilburg University
-
Affiliation as printed
University of Amsterdam
Cited by 1 stored of 1
1 result
No patents citing this paper on Lens.org (checked 2026-10-11).