Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints.

Abstract

Robust scientific knowledge is contingent upon replication of original findings. However, replicating researchers are constrained by resources, and will almost always have to choose one replication effort to focus on from a set of potential candidates. To select a candidate efficiently in these cases, we need methods for deciding which out of all candidates considered would be the most useful to replicate, given some overall goal researchers wish to achieve. In this article we assume that the overall goal researchers wish to achieve is to maximize the utility gained by conducting the replication study. We then propose a general rule for study selection in replication research based on the *replication value* of the set of claims considered for replication. The *replication value* of a claim is defined as the maximum expected utility we could gain by conducting a replication of the claim, and is a function of (1) the value of being certain about the claim, and (2) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using *replication value* as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of *replication value*. Our goal in this article is not to define concrete guidelines for study selection, but to provide the necessary theoretical foundations on which such concrete guidelines could be built.

Authors 12

1. Peder Mortvedt Isager

   Eindhoven University of Technology

   Affiliation as printed

   Department of Industrial Engineering & Innovation Sciences, Eindhoven University of Technology

2. Robbie C. M. van Aert

   Tilburg University

   Affiliation as printed

   Department of Methodology and Statistics, Tilburg University

3. Štěpán Bahník

   Prague University of Economics and Business

   Affiliation as printed

   Faculty of Business Administration, Prague University of Economics and Business

4. Mark John Brandt

   Tilburg University · Michigan State University

   Affiliation as printed

   Department of Psychology, Michigan State University

   Department of Social Psychology, Tilburg University

5. K. Andrew DeSoto

   Association for Psychological Science

   Affiliation as printed

   Association for Psychological Science

6. Roger Giner‐Sorolla

   University of Kent

   Affiliation as printed

   School of Psychology, University of Kent

7. Joachim Israel Krueger

   Brown University

   Affiliation as printed

   Department of Cognitive, Linguistic & Psychological Sciences, Brown University

8. Marco Perugini

   University of Milano-Bicocca

   Affiliation as printed

   Dipartimento di Psicologia, University of Milano-Bicocca

9. Ivan Ropovik

   University of Prešov · Charles University

   Affiliation as printed

   Faculty of Education, Charles University

   Faculty of Education, University of Presov

10. Anna van 't Veer Aachen Methodology and Statistics Unit Institute of Psychology, Leiden University

    Leiden University

    Affiliation as printed

    Methodology and Statistics unit, Institute of Psychology, Leiden University

11. Marek Albert Vranka

    Charles University

    Affiliation as printed

    Faculty of Social Sciences, Charles University

12. Daniël Lakens

    Eindhoven University of Technology

    Affiliation as printed

    Department of Industrial Engineering & Innovation Sciences, Eindhoven University of Technology