Replication value as a function of citation impact and sample size

Abstract

Researchers seeking to replicate original research often need to decide which of several relevant candidates to select for replication. Several strategies for study selection have been proposed, utilizing a variety of observed indicators as criteria for selection. However, few strategies clearly specify the goal of study selection and how that goal is related to the indicators that are utilized. We have previously formalized a decision model of replication study selection in which the goal of study selection is to maximize the expected utility gain of the replication effort. We further define the concept of replication value as a proxy for expected utility gain (Isager et al., 2020). In this article, we propose a quantitative operationalization of replication value. We first discuss how value and uncertainty - the two concepts used to determine replication value – could be estimated via information about citation count and sample size. Second, we propose an equation for combining these indicators into an overall estimate of replication value, which we denote RVCn. Third, we suggest how RVCn could be implemented as part of a broader study selection procedure. Finally, we provide preliminary data suggesting that studies that were in fact selected for replication tend to have relatively high RVCn estimates. The goal of this article is to explain how RVCn is intended to work and, in doing so, demonstrate the many assumptions that should be explicit in any replication study selection strategy.

Authors 3

1. Peder Mortvedt Isager
   Affiliation as printed

   Department of Psychology , Oslo New University College

   Lovisenberggata 13 , 0456 Oslo , Norway

   Department of Psychology, Oslo New University College

2. Anna van 't Veer Aachen Methodology and Statistics Unit Institute of Psychology, Leiden University

   Leiden University

   Affiliation as printed

   Methodology and Statistics unit , Institute of Psychology , Leiden University

   Methodology and Statistics unit, Institute of Psychology, Leiden University

3. Daniël Lakens

   Eindhoven University of Technology

   Affiliation as printed

   Department of Industrial Engineering & Innovation Sciences , Eindhoven University of Technology

   Department of Industrial Engineering & Innovation Sciences, Eindhoven University of Technology